Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Semantic Layers Matter (and how to build one with DuckDB)

motherduck.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#21

Post by anon84873628 »

sschnei8 wrote: post #6

I love a semantic layer as much as the next guy... Pivoting a decent sized BI shop toward using one instead of splashing the same SQL all over the place is *tough*. It's one of those: "the analyst could have been building important report for director and you want them to create re-usable logic??? we'll do that later, get report done now. Just copy/paste that SQL over here" This is how you end up with the the 1000 mo…

That tracks. The semantic layer is like a capital investment that pays off over time. So it can be hard to justify the initial investment if people don't grok the payoff.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#22

Post by mousematrix »

hey all, another perspective that I have been thinking about is if semantic layers are like ORM for but BI dashboards. Actually, they I think its more than BI dashboards since a similar idea applies to Features. Features in ML land are nothing but a Measure + Entity metadata + TTL. So, really its about higher-order semantics and as we move up the stack, we need richer expression to describe our world.

Feature stores explored here: https://www.xorq.dev/blog/featurestore-to-featurehouse

I think my key takeaway building this is that we need better expression systems and Ibis is a great foundation to build yours..maybe you want to build a language for some other domain etc.

PS: I am one of the authors of bsl and co-founder of Xorq.

I am one of the authors of bsl and founder of xorq.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#23

Post by sschnei8 »

dietr1ch wrote: post #14
halfcat wrote: post #7

Earlier quoted context omitted.

The flip side is, you often don’t know what needs to be reusable until you’ve had some iterations. Wrong abstractions can be way worse, and also gain their own momentum.

The problem is that often these quick or maybe not reusable are written in such a haste that there's no breadcrumbs left to do the right thing whenever you are done getting that urgent thing out (most likely never because "everything is urgent" :( )

Exactly. “Everything is urgent” is the great burden of our time lol

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#24

Post by seedless-sensat »

btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

Came to make the same comment. I got through like 12 paragraphs, and it still hadn't explained what a Semantic Layer is, so I gave up

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#25

Post by 12ian34 »

aszen wrote: post #12

I like the idea of a semantic Layer but don't think defining it in yaml is the right way to go about it. Semantic Layer needs proper language and tooling support which Malloy provides.

are you using Malloy in prod? Do you know of anyone? I remember the Malloy hype during Data Council in 2023. Haven't heard about it since.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#26

Post by articsputnik »

btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

From the article:

> There's a lot of information out there, including from myself about the history and rise [2022], comparing it to an MVC-like approach, or explaining its capabilities. That's why in this article I focus on the why and showcase how to use it in a practical example in the next chapter.

[1] https://www.ssp.sh/blog/rise-of-semantic-layer-metrics/ [2] https://cube.dev/blog/exploring-the-semantic-layer-through-t... [3] https://cube.dev/blog/universal-semantic-layer-capabilities-...

My one line definition that I use atm:

> A semantic layer acts as an intermediary, translating complex data into understandable user business concepts. It bridges the gap between raw data in databases (such as sales data with various attributes) and actionable insights (such as revenue per store or popular brands). This layer helps business users access and interpret data using familiar terms without needing deep technical knowledge. https://www.ssp.sh/brain/semantic-layer#semantic-layer-defin...

Edit: I'm the OP.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#27

Post by articsputnik »

Frotag wrote: post #16
cryptonector wrote: post #9

Is a "semantic layer" nothing more than a fancy name for a SQL VIEW in a NoSQL?

Kind of annoying the article writes "What is [a semantic layer] anyway?" twice but never defines it directly.

OP here - I wrote extensively about, that's why I linked to existing article rather than explaining once more, and focusing on the why and how to build one. See also comment above: https://news.ycombinator.com/reply?id=44960004&goto=item%3Fi...

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#28

Post by articsputnik »

aszen wrote: post #12

I like the idea of a semantic Layer but don't think defining it in yaml is the right way to go about it. Semantic Layer needs proper language and tooling support which Malloy provides.

Yeah, we still have them, MDX before and now DAX :)

I curate some more on here in case of interest: https://www.ssp.sh/brain/data-modeling-languages.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#29

Post by porridgeraisin »

CharlesW wrote: post #11
cryptonector wrote: post #9

Is a "semantic layer" nothing more than a fancy name for a SQL VIEW in a NoSQL?

It's a lot more. A SQL VIEW is just a saved query, where a semantic layer defines the shared meaning of the data, and helps enforce consistent metrics, joins, and logic across tools. You'd be surprised at how many ways "active customer" can be represented as SQL.

Doesn't a view do that?

  create view active_cx as select * from customer join audit_events using(...) join ... where -- active condition

  -- use active_cx wherever

  select ... from orders join active_cx using(...) where ts > start_of_month() group by active_cx.id

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#30

Post by hugh-avherald »

articsputnik wrote: post #26
btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

From the article: > There's a lot of information out there, including from myself about the history and rise [2022], comparing it to an MVC-like approach, or explaining its capabilities. That's why in this article I focus on the why and showcase how to use it in a practical example in the next chapter. [1] https://www.ssp.sh/blog/rise-of-semantic-layer-metrics/ [2] https://cube.dev/blog/exploring-the-semantic-layer-t…

I don't think this qualifies as a definition.
Post reply on HN