Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Semantic Layers Matter (and how to build one with DuckDB)

motherduck.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#31

Post by refset »

btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

Julian Hyde (Apache Calcite, Google) gave a crisp presentation on this and how SQL could express 'measures' to bridge the gap: https://communityovercode.org/wp-content/uploads/2023/10/mon...

> A semantic layer, also known as a metrics layer, lies between business users and the database, and lets those users compose queries in the concepts that they understand. It also governs access to the data, manages data transformations, and can tune the database by defining materializations.

There's also now a paper: https://arxiv.org/pdf/2406.00251

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#32

Post by Bjartr »

articsputnik wrote: post #26
btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

From the article: > There's a lot of information out there, including from myself about the history and rise [2022], comparing it to an MVC-like approach, or explaining its capabilities. That's why in this article I focus on the why and showcase how to use it in a practical example in the next chapter. [1] https://www.ssp.sh/blog/rise-of-semantic-layer-metrics/ [2] https://cube.dev/blog/exploring-the-semantic-layer-t…

That's what it does, but that doesn't tell us what it is.

Defining a car as "a vehicular conveyance that helps people get from A to B" is similarly technically correct, but provides little help to the reader in determining if the thing they're looking at is a car or not.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#33

Post by efromvt »

sschnei8 wrote: post #6

I love a semantic layer as much as the next guy... Pivoting a decent sized BI shop toward using one instead of splashing the same SQL all over the place is *tough*. It's one of those: "the analyst could have been building important report for director and you want them to create re-usable logic??? we'll do that later, get report done now. Just copy/paste that SQL over here" This is how you end up with the the 1000 mo…

Yeah, minimizing the gap between the semantic layer authoring and adhoc is what you need to do to close that - there has to be a progressive model both for consumption (take this semantic layer, slightly extend/tweak it in an adhoc fashion) and for organically promoting up the adhoc works to the layer.

Right now a lot of semantic tools introduce a big discontinuity in both workflows that keeps the two worlds separate.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#34

Post by efromvt »

articsputnik wrote: post #28
aszen wrote: post #12

I like the idea of a semantic Layer but don't think defining it in yaml is the right way to go about it. Semantic Layer needs proper language and tooling support which Malloy provides.

Yeah, we still have them, MDX before and now DAX :) I curate some more on here in case of interest: https://www.ssp.sh/brain/data-modeling-languages .

MDX really did it well, at the cost of being impenetrable to the average user - the drilldown flow you could get is still hard to beat.

Shameless plug for the list, though - I work on https://github.com/trilogy-data/pytrilogy - semantic layer directly embedded in otherwise (mostly) SQL syntax.

I'll do an equivalent example on the taxi dataset when I have some time.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#35

Post by Bjartr »

articsputnik wrote: post #26
btbuildem wrote: post #18

Impressive! An entire article about semantic layers, artfully avoids ever defining what a semantic layer is. Let me take a swipe at it: a semantic layer helps express queries and their results in terms the end-consumers will care about / prefer to reason in, instead of whatever extremely correct and efficient atrocities the database nerds came up with. Did I get that right?

From the article: > There's a lot of information out there, including from myself about the history and rise [2022], comparing it to an MVC-like approach, or explaining its capabilities. That's why in this article I focus on the why and showcase how to use it in a practical example in the next chapter. [1] https://www.ssp.sh/blog/rise-of-semantic-layer-metrics/ [2] https://cube.dev/blog/exploring-the-semantic-layer-t…

I started writing another comment and after looking at your links and was about to quote the definition at the top of "The Rise of the Semantic Layer" as a suggestion, but I realized that it actually isn't that far off, information-wise from the definition youve provided here.

So I took a step back and tried to think about why one "feels" to a reader more like a definition than the other. I think it comes down to phrasing more than informational content. The definition you provide in your comment comes off, for lack of a better term, too much like a sales pitch.

Less is more when it comes to definitions, at least for defining terms in articles/blog posts like these.

Here's my attempt at a better (for this use case) definition:

A semantic layer is an interface to data stores that is designed to be queryable in terms relevant and familiar to those with knowledge of the business domain.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#36

Post by apwell23 »

12ian34 wrote: post #25
aszen wrote: post #12

I like the idea of a semantic Layer but don't think defining it in yaml is the right way to go about it. Semantic Layer needs proper language and tooling support which Malloy provides.

are you using Malloy in prod? Do you know of anyone? I remember the Malloy hype during Data Council in 2023. Haven't heard about it since.

i think the team moved on from google ( which was the main benefactor)

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#37

Post by cryptonector »

articsputnik wrote: post #27
Frotag wrote: post #16

Earlier quoted context omitted.

Kind of annoying the article writes "What is [a semantic layer] anyway?" twice but never defines it directly.

OP here - I wrote extensively about, that's why I linked to existing article rather than explaining once more, and focusing on the why and how to build one. See also comment above: https://news.ycombinator.com/reply?id=44960004&goto=item%3Fi...

I looked for such a link in TFA, and it wasn't obvious.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#38

Post by secondrow »

mritchie712 wrote: post #15
sschnei8 wrote: post #6

I love a semantic layer as much as the next guy... Pivoting a decent sized BI shop toward using one instead of splashing the same SQL all over the place is *tough*. It's one of those: "the analyst could have been building important report for director and you want them to create re-usable logic??? we'll do that later, get report done now. Just copy/paste that SQL over here" This is how you end up with the the 1000 mo…

this is why I liked Looker. The only way to build reports was from the semantic layer which was easy to use and built into the BI. we took the same approach when we started https://www.definite.app/ .

This is why I liked BusinessObjects

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#39

Post by secondrow »

mousematrix wrote: post #22

hey all, another perspective that I have been thinking about is if semantic layers are like ORM for but BI dashboards. Actually, they I think its more than BI dashboards since a similar idea applies to Features. Features in ML land are nothing but a Measure + Entity metadata + TTL. So, really its about higher-order semantics and as we move up the stack, we need richer expression to describe our world. Feature stores…

I’m familiar with Xorq. One of features of the Xorq library that I find interesting is that it catalogs data processing (compute) expressions as it compiles, along with call lineage. Makes reuse easier for SQL and non-SQL processing.

Re: Why Semantic Layers Matter (and how to build one with DuckDB)

#40

Post by secondrow »

secondrow wrote: post #39
mousematrix wrote: post #22

hey all, another perspective that I have been thinking about is if semantic layers are like ORM for but BI dashboards. Actually, they I think its more than BI dashboards since a similar idea applies to Features. Features in ML land are nothing but a Measure + Entity metadata + TTL. So, really its about higher-order semantics and as we move up the stack, we need richer expression to describe our world. Feature stores…

I’m familiar with Xorq. One of features of the Xorq library that I find interesting is that it catalogs data processing (compute) expressions as it compiles, along with call lineage. Makes reuse easier for SQL and non-SQL processing.

FYI - https://github.com/xorq-labs/xorq
Post reply on HN