Launch HN: Narrator (YC S19) – a data modeling platform built on a single table

Appreciate the skepticism, since yes, we're a totally different approach.

I'll try to address your points in order.

Yes, we agree that 99% of the work is determining what the data means. Our structure doesn't magically make things better because of its structure. It's that once you have it analysis / aggregation on top of it becomes substantially less work because you don't have to constantly redo models to answer new questions.

Yeah, we've all been there -- production DBs aren't typically architected to store historical data. But we've found in practice that the data sources you most care about do have it. Page views, emails sent / received, completed orders, etc. all have timestamps. And for some things you don't need it. If you wanted to do a query with all customers who are VIPs, you wouldn't need a 'became VIP' activity. Adding is_VIP as a feature to the customer in the activity stream works too. Generally if you can do an analysis the more traditional way then you should already have the data to do it in Narrator too.

Sure, star schemas are the way of doing things and this is a new approach. But the efficiency gains realized by our own data scientists are enough to where they wouldn't go back -- it warrants the investment in learning it. Our challenge as a business will be how to convince others of that as well.

What we mean by single source of truth is that data is internally consistent - each term is defined once. In your scenario you'll have a single 'completed order' activity with the total order amount. If you want to add shipping cost that's fine -- add a 'added shipping to order' activity with the cost in it. Do the same with sales tax. Bob, Alice, and Jimmy can create reports with whatever activities they want. The crucial point is by making those reports they're not defining a new model. They're just combining activities. Since all tables are generated straight from the activity stream a future analyst won't use a materialized view based on Bob's data to build a new report -- they'll build it straight from the original activities.

Definitions are super important. Getting agreement on definitions is really hard and can easily get us stuck in a loop.

In my last job, we spent months talking about "What is a Sale?" is it when someone signs a contract or when they pay or when they move in, etc...

Then you add your sale metric into a table and as that table is used in different places the sales numbers till don't match, people don't remember what a sale is and the conversation starts up again.

Why is Narrator different?

In Narrator, you don't define what a sale is, you break up your data into customer actions. So "Signed Contract", "Paid Invoice", "Moved In" and thus as people ask different questions we can alway clearly see what they are referring to.

Step 1 to alignment is guaranteeing consistency and transparency.

How do you deal with states? You are right that does make it really hard. We see customers leveraging the incremental aspect of the activity stream to diff the stream with the updated_at to pull out changes as the activity stream updates. (every 15 minutes so yes you will loose changes in between that time). This doesn't solve the problem but does take you much closer. And then when you do add proper timestamps then you can have historical data in the activity stream and cleaned data from you new tables merged. All the users using that activity are NOT affected.

Not perfect but allows us to have as accurate data as possible.

What about the market? Yes, I agree the market is a challenge since the world has been only using star schema. We hope that standardization, speed, reusability aspect of Narrator is so compelling that people slowly switch.

How do we get numbers to match? Consistency and Transparency. Every one who uses the "Completed Order" uses the same revenue. SO it is consistent! Then you can add the "Shipped Order" activity which has the shipping amount.

By having clear CONSISTENT definitions and clear activities you end up in a world where your numbers match. The only ways for numbers not to match is if some is being deliberate about getting the data not to match which thanks to dataset, is always transparent.

Definitions are a social problem but technology, limitations and consistency can help a lot.

scheduled!

Thanks. Interesting approach!

Yeah our first iteration of the activity stream was at WeWork and it was impossible to use. For this reason we built our Dataset tool and thanks to our innovation of relationships we are now able to make use of this structure to create any table.

Often, people think that just creating a time-series table is enough but it is so hard to use and so hard to maintain that you will hate yourself. Narrator solve all those problems so your experience become truly incredible!