Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

21–30 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#21

Post by jacques_chester »

pedalpete wrote: post #19
luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

I had the same response. So reading what you've posted here, it appears to be a smart(er) cache. Do you think it's valuable to have this as a service rather than roll your own solution?

I think the main step forward is that it has an efficient means for calculating what views change when a new datum arrives. Which means that it could, in theory, hold a very large amount of views for relatively low overhead.

It's a surprisingly difficult problem. I wouldn't roll my own.

Re: Materialize: A Streaming Data Warehouse

#22

Post by Darkphibre »

slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

I architected and implemented a true-realtime telemetry pipeline. The requirement was subsecond per-user aggregation and round-trip notification of thresholds exceeded. Took us a couple years, but when Halo 5 launched, we handled 2.5B events/hour without breaking a sweat (AMQP over Websockets). It's since been rolled out to multiple Microsoft 1st-party games.

The round-trip requirement was dropped before we launched, reducing the usage of the technology stack to pure telemetry gathering.

The analysts are all perfectly happy with 5-10 minute delays.

Link to my GDC talk, in case people are interested: https://www.youtube.com/watch?v=o098roxWAkA

Re: Materialize: A Streaming Data Warehouse

#24

Post by jacques_chester »

frankmcsherry wrote: post #13
polskibus wrote: post #10

Earlier quoted context omitted.

Thanks. What happens when a DDL is run on the source table (add or remove column for instance) ? Does it break the entire flow?

DDLs that alter schemas definitely cause things to catch on fire at the moment (Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics). In principle `CREATE` and `DROP` statements are things we could pass through, but some work needs to get done to pull the right information out of Debezium at the right moments (and drop the right prefixes of the existing Kafka topics if you re…

> Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics

That's unfortunate. It seems like it would be ideal to feed schema changes through the same topic, so that the ordering is defined without the usual distributed-system hilarity.

Re: Materialize: A Streaming Data Warehouse

#25

Post by benesch »

jacques_chester wrote: post #21
pedalpete wrote: post #19

Earlier quoted context omitted.

I had the same response. So reading what you've posted here, it appears to be a smart(er) cache. Do you think it's valuable to have this as a service rather than roll your own solution?

I think the main step forward is that it has an efficient means for calculating what views change when a new datum arrives. Which means that it could, in theory, hold a very large amount of views for relatively low overhead. It's a surprisingly difficult problem. I wouldn't roll my own.

Yes, absolutely. Entire dissertations have been written on the topic: https://dash.harvard.edu/bitstream/handle/1/14226097/KATE-DI...

Efficiently maintaining views over arbitrarily complex computations is one of the two hard problems in computer science for a reason [0].

[0]: https://martinfowler.com/bliki/TwoHardThings.html

Re: Materialize: A Streaming Data Warehouse

#26

Post by TeMPOraL »

je42 wrote: post #20
slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

I think it would be helpful if you could dive deeper why you think " Refreshing the data every five minutes in batches" is "sufficient". From my perspective: batching is more complicated, than batching. (Batching requires you to define parameters like batch size and interval, while streaming does not for example). But may be batching tools are simpler than streaming tools, but i am not so sure. Batching in general ha…

> I think it would be helpful if you could dive deeper why you think " Refreshing the data every five minutes in batches" is "sufficient".

Not OP, but I'm guessing because most of that data is not actionable in real-time. There's zero point to get real-time data to analysts or decision makers if they're not going to use it to make real-time decisions; arguably, it can be even counterproductive, leading to an organizational ADHD, where people fret over minute-to-minute changes, where they should be focusing on daily or monthly running averages.

Re: Materialize: A Streaming Data Warehouse

#27

Post by jacques_chester »

TeMPOraL wrote: post #26
je42 wrote: post #20

Earlier quoted context omitted.

I think it would be helpful if you could dive deeper why you think " Refreshing the data every five minutes in batches" is "sufficient". From my perspective: batching is more complicated, than batching. (Batching requires you to define parameters like batch size and interval, while streaming does not for example). But may be batching tools are simpler than streaming tools, but i am not so sure. Batching in general ha…

> I think it would be helpful if you could dive deeper why you think " Refreshing the data every five minutes in batches" is "sufficient". Not OP, but I'm guessing because most of that data is not actionable in real-time. There's zero point to get real-time data to analysts or decision makers if they're not going to use it to make real-time decisions; arguably, it can be even counterproductive, leading to an organiza…

While they focus on the very-fast-updates thing, I think their technology will apply to batch cases also. In either of streaming or batching I want to do the least possible work, their claim is that they can skip a lot of unnecessary computations automatically.

That said, I find that batch systems have enormous inertia due to simple don't-touch-it syndrome. A report got developed in 1992 for a manager who retired in 1998 and died in 2009. Each night it churns through 4 billion records in a twelve-way join that costs tens of thousands of dollars of computing time per year.

Who reads this report? Nobody. In fact, the person who asked for it read it two or three times and then stopped. But it's landed reliably in an FTP folder for 28 years and by god nobody is game to find out whether the CEO reads it religiously.

Re: Materialize: A Streaming Data Warehouse

#28

Post by jacques_chester »

benesch wrote: post #25
jacques_chester wrote: post #21

Earlier quoted context omitted.

I think the main step forward is that it has an efficient means for calculating what views change when a new datum arrives. Which means that it could, in theory, hold a very large amount of views for relatively low overhead. It's a surprisingly difficult problem. I wouldn't roll my own.

Yes, absolutely. Entire dissertations have been written on the topic: https://dash.harvard.edu/bitstream/handle/1/14226097/KATE-DI... Efficiently maintaining views over arbitrarily complex computations is one of the two hard problems in computer science for a reason [0]. [0]: https://martinfowler.com/bliki/TwoHardThings.html

Never let HN haters (among whom I am frequently numbered, it ought to be noted) get you down.

After skimming a bit of the differential dataflow writing I am really impressed. This is deep computer science doing what it does best, which is to do much more with much less.

Re: Materialize: A Streaming Data Warehouse

#29

Post by arjunnarayan »

jacques_chester wrote: post #24
frankmcsherry wrote: post #13

Earlier quoted context omitted.

DDLs that alter schemas definitely cause things to catch on fire at the moment (Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics). In principle `CREATE` and `DROP` statements are things we could pass through, but some work needs to get done to pull the right information out of Debezium at the right moments (and drop the right prefixes of the existing Kafka topics if you re…

> Debezium writes them to a side-topic, which isn't obviously synchronized with the data topics That's unfortunate. It seems like it would be ideal to feed schema changes through the same topic, so that the ordering is defined without the usual distributed-system hilarity.

Agreed! We plan to contribute to the Debezium project over the next months along these lines. We’ll have more to say on this shortly!

Re: Materialize: A Streaming Data Warehouse

#30

Post by benesch »

slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

> I just want to say this is a very dangerous assumption to make.

I think we're actually arguing the same points here. It's not that every use case needs single-digit millisecond latencies! There are plenty of use cases that are satisfied by batch jobs running every hour or every night.

But when you do need real-time processing, the current infrastructure is insufficient. When you do need single-digit latency, running your batch jobs every second, or every millisecond, is computationally infeasible. What you need is a reactive, streaming infrastructure that's as powerful as your existing batch infrastructure. Existing streaming infrastructure requires you to make tradeoffs on consistency, computational expressiveness, or both; we're rapidly evolving Materialize so that you don't need to compromise on either point.

And once you have streaming data warehouse in place for the use cases that really demand the single-digit latencies, you might as well plug your analysts and data scientists into that same warehouse, so you're not maintaining two separate data warehouses. That's what we mean by ideal: not only does it work for the systems with real-time requirements, but it works just as well for the humans with looser requirements.

To give you an example, let me respond to this point directly:

> Secondly, almost all data is useless in its raw form. The analysts had to perform ELT jobs on their data in the warehouse to clean, dedupe, aggregate, and project their business rules on that data. These functions often require the database to scan over historical data to produce the new materializations of that data. So even if we could get the data in the warehouse in sub-minute latency, the jobs to transform that data ran every 5 minutes.

The idea is that you would have your analysts write these ETL pipelines directly in Materialize. If you can express the cleaning/de-duplication/aggregation/projection in SQL, Materialize can incrementally maintain it for you. I'm familiar with a fair few ETL pipelines that are just SQL, though there are some transformations that are awkward to express in SQL. Down the road we might expose something closer to the raw differential dataflow API [0] for power users.

[0]: https://github.com/TimelyDataflow/differential-dataflow

Post reply on HN