Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

51–60 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#51

Post by benesch »

yomly wrote: post #40
luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

This reminds me a lot about Noria DB. Wonder if anyone familiar with both can shed any further light?

Indeed, Materialize is quite similar to Noria, and has the Frank McSherry stamp of awesomeness. [0] We know many of the Noria folks and have a lot of respect for them and their work. I also worked on the Noria project for a summer in college, and am a full-time engineer at Materialize now.

The biggest difference is one of intended use. Noria is, first and foremost, a research prototype, intended to explore new ideas in systems research. Materialize, by contrast, is intended to be a rock-solid piece of production infrastructure. (Much of the interesting research, in timely and differential dataflow, is already done.) We've invested a good bit in supporting the thornier bits of SQL, like full joins, nested subqueries, correlated subqueries, variable-precision decimals, and so on. Noria's support for SQL is less extensive; I think the decisions have been guided mostly by what's necessary to run its lobste.rs and HotCRP benchmarks. Make no mistake: Noria is an impressive piece of engineering, but, as an enterprise looking to deploy Noria, there's no one you can pay for support or to implement feature requests.

One area where Noria shines is in partial materialization. Details are in the Noria paper [1], but the tl;dr is that Noria has a lot of smarts around automatically materializing only the subset of the view that is actually accessed, while presently Materialize requires that you explicitly declare what subsets to materialize. We have some plans for how to bring these smarts to Materialize, but we haven't implemented them yet.

Also worth noting is that Materialize's underlying dataflow engine, differential dataflow, has the ability to support iterative computation, while Noria's engine requires an acyclic dataflow graph. We don't yet expose this power in Materialize, but will soon. Put another way: `WITH RECURSIVE` queries are a real and near possibility in Materialize, while (as I understand it) `WITH RECURSIVE` queries would require substantial retooling of Noria's underlying dataflow engine.

One of the creators of Noria, Jon Gjengset, did an interview on Noria [2] that covered some of differences between Noria and differential dataflow from his perspective, which I highly recommend you check out as well!

[0]: https://twitter.com/frankmcsherry/status/1056957760435376129...

[1]: https://jon.tsp.io/papers/osdi18-noria.pdf

[2]: https://notamonadtutorial.com/interview-with-norias-creator-...

Re: Materialize: A Streaming Data Warehouse

#52

Post by benesch »

benesch wrote: post #51
yomly wrote: post #40

Earlier quoted context omitted.

This reminds me a lot about Noria DB. Wonder if anyone familiar with both can shed any further light?

Indeed, Materialize is quite similar to Noria, and has the Frank McSherry stamp of awesomeness. [0] We know many of the Noria folks and have a lot of respect for them and their work. I also worked on the Noria project for a summer in college, and am a full-time engineer at Materialize now. The biggest difference is one of intended use. Noria is, first and foremost, a research prototype, intended to explore new ideas…

I forgot to mention: differential dataflow is capable of providing much stronger consistency guarantees than Noria is. Differential dataflow is consistency preserving—if your source of truth provides strict serializability, differential can also provide strict serializability—while Noria provides only eventual consistency.

Tapping into the consistency-preserving features in Materialize is a bit complicated at the moment, but we're actively working on improving the integration of our consistency infrastructure for MySQL and PostgreSQL upstreams.

Re: Materialize: A Streaming Data Warehouse

#53

Post by chimychang77 »

benesch wrote: post #45
vhold wrote: post #44

Earlier quoted context omitted.

I think what might be really unique here that people aren't imagining, are the new possible applications of having With sufficiently expressive SQL and UDF support there are whole classes of stateful services that are performing lookups, aggregations, etc, that could be written as just views on streams of data. Experts who model systems in SQL, but aren't experts in writing distributed stateful streaming services wou…

I agree wholeheartedly with your take! Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support in a streaming system, as you have to reconstruct the potentially large window whenever you receive new data. Probably some interesting research to be done here, or at least some…

I think the average data team at a startup using something like Redshift/BigQuery/Snowflake is using window functions quite extensively when writing analytical queries/building data pipelines so I'm surprised to hear you haven't see much customer demand for them.

If you trying to wholesale replace a "traditional" batch oriented data warehouse like the ones I mentioned above I think building support for window functions would be essential.

Re: Materialize: A Streaming Data Warehouse

#54

Post by benesch »

jacques_chester wrote: post #50
benesch wrote: post #45

Earlier quoted context omitted.

I agree wholeheartedly with your take! Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support in a streaming system, as you have to reconstruct the potentially large window whenever you receive new data. Probably some interesting research to be done here, or at least some…

I have a strong suspicion that bitemporalism makes a lot of these problems less problematic. The actual volumes of data are the same, but the all-or-nothingness of windowing over very large data sets in order to avoid missing anything that arrived late goes away. I wrote shambolic stream-of-consciousness notes on it several years ago: https://docs.google.com/document/d/1ZlPp099_fV1lyYWACSyuWY_j... The gist being that…

We’re thinking along very similar lines! We’ve got some of our thoughts around evolving how timestamps work in Materialize written down here: https://github.com/MaterializeInc/materialize/issues/1309

Re: Materialize: A Streaming Data Warehouse

#55

Post by benesch »

chimychang77 wrote: post #53
benesch wrote: post #45

Earlier quoted context omitted.

I agree wholeheartedly with your take! Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support in a streaming system, as you have to reconstruct the potentially large window whenever you receive new data. Probably some interesting research to be done here, or at least some…

I think the average data team at a startup using something like Redshift/BigQuery/Snowflake is using window functions quite extensively when writing analytical queries/building data pipelines so I'm surprised to hear you haven't see much customer demand for them. If you trying to wholesale replace a "traditional" batch oriented data warehouse like the ones I mentioned above I think building support for window functio…

To be clear, window functions are definitely on our radar! But you’d be surprised how many folks are delighted just to have more basic SQL features, like fully functional joins. Streaming SQL is surprisingly far behind batch SQL.

Re: Materialize: A Streaming Data Warehouse

#58

Post by airstrike »

pedalpete wrote: post #19
luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

I had the same response. So reading what you've posted here, it appears to be a smart(er) cache. Do you think it's valuable to have this as a service rather than roll your own solution?

This comment reminded me of this classic: https://news.ycombinator.com/item?id=9224

Re: Materialize: A Streaming Data Warehouse

#59

Post by Qwuke »

pedalpete wrote: post #19
luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

I had the same response. So reading what you've posted here, it appears to be a smart(er) cache. Do you think it's valuable to have this as a service rather than roll your own solution?

The differential dataflow codebase was really polished and optimized last I saw - when they say "demand milliseconds" I think they put the effort into delivering that.

Additionally, given it will be Apache licensed in 4 years, I think it'd be good to ask "Can I wait?" before going all in

Re: Materialize: A Streaming Data Warehouse

#60

Post by jacques_chester »

benesch wrote: post #54
jacques_chester wrote: post #50

Earlier quoted context omitted.

I have a strong suspicion that bitemporalism makes a lot of these problems less problematic. The actual volumes of data are the same, but the all-or-nothingness of windowing over very large data sets in order to avoid missing anything that arrived late goes away. I wrote shambolic stream-of-consciousness notes on it several years ago: https://docs.google.com/document/d/1ZlPp099_fV1lyYWACSyuWY_j... The gist being that…

We’re thinking along very similar lines! We’ve got some of our thoughts around evolving how timestamps work in Materialize written down here: https://github.com/MaterializeInc/materialize/issues/1309

Sort of -- the problem I see in the event time / processing time distinction is that it's about instants rather than intervals. There are a number of models and queries that are not reliably expressible with instants alone, unless you reinvent intervals with them.

For example, if I rely on "updated-at" and infer that whatever record has the latest updated-at is the "current" record, then I may create the illusion that there are no gaps in my facts. That may not be so.

A reference system to look at is Crux: https://opencrux.com/

Post reply on HN