Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 61 posts

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#21

Post by MechanicalTwerk »

ahmedNarrator wrote: post #8
MechanicalTwerk wrote: post #4

Wish you the best of luck but isn't this just a fancier version of EAV? https://en.wikipedia.org/wiki/Entity%E2%80%93attribute%E2%80... IMO, it doesn't matter what kind of db technology, schema or query tool that you use. A company will always have analysis sprawl regardless of whether those analyses are represented as data lake files, sql tabes, materialized views, regular views, or (as is often the case with EAV an…

Yeah, Entity modeling was one of the big inspirations to our approach. The main difference is how do you reassemble the single time-series table to create any table. This was quite a challenge and I think what makes the traceability and source of truth problem a lot simpler. In Narrator, the data team writes small SQL to create single customer centric business concepts that we call activities. These are around 25 lin…

> activities and how they relate

This is the problem with EAV/nosql/schemaless/etc and ultimately the problem I think you are going to have to solve. Instead of using ETL to model how the activities relate and reifying that model as database objects, EAV just kicks the can down the road to the query/BI tool.

Sprawl - The BI tool will end up containing most of the real business logic sprawled across many reports.

Single source of truth - A lot of the reports will be very similar but they will be based off slightly different activities or slightly different filtering logic. Which report is the correct one?

Traceability - I think this is more of an end-to-end "garbage-in, garbage-out" problem that all ETL/BI tools have that wouldn't be specific to your tool. It's more of an organizational/people problem.

Coherent model - In my experience, EAV isn't enough to cover the breadth of analyses mature businesses need to do and most business users won't be able to wrap their head around it. There will have to be some data person that creates a more coherent, tabular/spreadsheet-like model and in the case of this tool it looks like that model will have to exist in the BI tool. Which brings us back to sprawl/single source of truth issues.

Just some thoughts. But always glad to see more people working on stuff like this!

Edit - one last thing I wanted to mention. I think in reality you are going to find it takes more than ~25 lines of sql to define activities. That may be the case if the source is a schema that gets spit out of something like Stitch, but many other schemas in the wild will take a lot more than 25 loc to massage into your 11 column schema.

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#22

Post by ahmedNarrator »

whalesalad wrote: post #18

This approach sounds very similar to using a single table in DynamoDB: https://www.alexdebrie.com/posts/dynamodb-single-table/

Yeah DynamoDB is very much aligned with a single table. We do it in the customers warehouse so they have access to all their data.

The real magic is how do you assemble the data into tables that can answer any question when the output table is billions of rows. Unfortunately, that requires a lot of SQL magic that depends on taking advantage of a columnar warehouse.

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#24

Post by slotrans »

Hard to imagine a worse idea.

The problem in data warehousing is not the structure, it's the definitions. It doesn't matter one tiny little bit whether you store your data in a fact/dimension star schema, a normalized OLTP-style schema, arbitrary aggregate/rollup tables, or this one-table monstrosity you've constructed. You still have to do the work of determining what the data means.

This isn't 50% of the work of building a data warehouse. It's 99% of the work.

This "Activity Stream" concept has some merit but most data sources can't support it. Much to my eternal frustration, most systems only store current state, and have no concept of history or events. You can't extract "activities" from a data system that doesn't record them.

Still, maybe I'm wrong and your idea is great. The market will reject it.

Star schema modeling is woven into the very fabric of the data warehousing profession. We use it everywhere, whether it's appropriate or not (and it is often not appropriate). It's what all the BI tools support. It's what analysts know how to write queries against. Your thing may be cool and all but no one knows how to use it.

One last thing: "Numbers would always match (single source of truth)..."

Not even close. This belies just massive ignorance of how analytics actually work. Let's say I have a data point that says the total amount of order #123 is $42.00. Ok... does that include shipping? Sales tax? Oh it does, and finance cares about that, so now I need to decompose that into 34.95 + 5.00 + 2.05 and label appropriately. Now Bob's report can sum the item total (no shipping, no taxes) and Alice's report can include shipping but not taxes, and Jimmy's report can sum the total, and now your wonderful "single source of truth" story goes in the toilet. Because "$42.00" isn't truth it's just a value. What turns it into truth is definition and consensus, which are social problems, not technical problems.

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#25

Post by texasbigdata »

So cool! Had a weekend tinker a few weeks ago that needed this use case and a quick search didn’t produce any useable light solutions so did it by hand.

One tiny data point for y’all.

Good luck!

Edit: assuming eventually your schema transformation unlocks at least partially and you have at least some flexibility outside the default 11 column approach?

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#26

Post by cedricd »

texasbigdata wrote: post #25

So cool! Had a weekend tinker a few weeks ago that needed this use case and a quick search didn’t produce any useable light solutions so did it by hand. One tiny data point for y’all. Good luck! Edit: assuming eventually your schema transformation unlocks at least partially and you have at least some flexibility outside the default 11 column approach?

Great to hear from someone who also built this themselves!

As far as flexibility beyond 11 columns: I'd love to know your use case.

We do support additional metadata on each activity with what we call enrichment tables.

Some events are going to need more metadata -- a page view would want to have the actual page, the five UTM parameters, referrer, etc, which is more than the 3 fields of metadata we store on the activity stream.

So we also support creating additional tables to add metadata to each activity. Each row requires a unique activity id and its timestamp and can an unlimited number of additional columns.

We'll then automatically join that table into the activity stream when queries need it.

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#27

Post by chasers »

cedricd wrote: post #16
sroussey wrote: post #5

What are the 11 columns?

Also replying since I wrote this up :) - activity_id : a unique identifier for the row - activity : the type of activity (eg 'page_view') - timestamp : time the activity happened - customer : the unique customer identifier Metadata columns Three columns for any info we'd like to add to an activity. Eg for a purchased product activity it could be product name. - feature_1 - feature_2 - feature_3 - revenue_impact : the…

Only three metadata fields or can you have more?

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#28

Post by cedricd »

dksidana wrote: post #23

Are you trying to build product analytics usecase like mixpanel/amplitude with this as well ?

No, we mainly focus on data modeling on top of a warehouse and deep analyses. Our customers are often data analysts, data scientists, or engineers.

That being said we would love to partner with a CDP like mixpanel and amplitude to have marketers and product people get quick insights using the data that is modeled and cleaned by the data team.

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#29

Post by ahmedelsama »

MechanicalTwerk wrote: post #21
ahmedNarrator wrote: post #8

Earlier quoted context omitted.

Yeah, Entity modeling was one of the big inspirations to our approach. The main difference is how do you reassemble the single time-series table to create any table. This was quite a challenge and I think what makes the traceability and source of truth problem a lot simpler. In Narrator, the data team writes small SQL to create single customer centric business concepts that we call activities. These are around 25 lin…

> activities and how they relate This is the problem with EAV/nosql/schemaless/etc and ultimately the problem I think you are going to have to solve. Instead of using ETL to model how the activities relate and reifying that model as database objects, EAV just kicks the can down the road to the query/BI tool. Sprawl - The BI tool will end up containing most of the real business logic sprawled across many reports. Sing…

Sprawl - YES! I would never put a single time-series table in your BI tool. It is not queryable and you will hate the insane results.

- We actually built our own query layer called Dataset to make sure that the dataset is materialized. This way if you put it in your BI tool, you can always go back to the dataset which points direct to the activity stream.

Single Source of truth. & traceability - 100%. We really aim to have activities be actually different. Each activity is modeled via SQL and often done by a data engineer or analyst. You cannot just create 1000 activities. 90% of our customers have between 20-40. This enables your activities to be unique. Also unlike tables, activities are building block so they map to something real (i.e. "paid invoice", "sent contract").

So far we haven't seen many people struggling with activities being too similar.

Also the modeling of the activity helps clear up the Garbage in -> Garbage out problem that often happens with CDPs (mixpanel, segment, etc..).

In terms of analysis. We did build a tool called Narrative (actionable analysis in a story format). This is designed to get users to write their analysis with CONTEXt built in vs just numbers on a screen. With context + the ability to click to see the activities and relationship people can quickly know what data powers the source. Does this solve the problem 100%? Nope, but it does take us huge steps in the right direction.

Coherent Model - I think our tool Dataset helps with this problem. We started as a consultancy and answered 1000s of questions over 3 years till our tool was able to answer any question. I usually demo by asking the customer to ask any question they have and I try to answer it live. So far, we have been able to answer them all so I am SUPER excited to find the limit of our tools.

Yeah, for data EL via Stitch, Fivetran then this is easy. Dirty data that is a bunch of JSONS etc... take a bit more effort but that building of the activity is done once. You also don't have to deal with how concepts relate or identity resolution or a lot of other things that make SQL complex.

Overall, I love this conversation and would like to continue. I am excited to hear some of your edge cases. Maybe we can even setup some time and talk face to face: https://calendly.com/ahmed-narrator/30min-1

Re: Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

#30

Post by dang »

slotrans wrote: post #24

Hard to imagine a worse idea. The problem in data warehousing is not the structure , it's the definitions . It doesn't matter one tiny little bit whether you store your data in a fact/dimension star schema, a normalized OLTP-style schema, arbitrary aggregate/rollup tables, or this one-table monstrosity you've constructed. You still have to do the work of determining what the data means . This isn't 50% of the work of…

Can you please make your substantive points thoughtfully, rather than in the flamewar style? This is particularly important when people are sharing their work. We don't want a culture in which people get flamed and belittled for doing that.

This is important because the effects that comments like this one produce are much stronger than the people making them assume they are. Worse, they compound. Then, unintentionally, we end up with an asshole culture which no one would want to subject themselves to. On HN, we want the incentives to go exactly the other way. I'm sure you can make your substantive criticisms without putdowns and name-calling if you want to—and that would actually be a quite valuable contribution.

https://news.ycombinator.com/newsguidelines.html

Post reply on HN