The next generation of Materialize • Hacker News Board

Re: The next generation of Materialize

#31

Post by benesch » Mon, Oct 03, 2022, 6:06 PM UTC

etherealpine wrote: post #22

Does this fix how much of an insane memory hog materialize is? Some queries are just impossible if you can’t use disk. This is why I was forced to stick with Flink. Even though materialize makes things appear stupid simple and easy with SQL, I found that you can only do the most simple streaming views with it. You can’t even do unique counts with this for very long without breaking—and there’s no probabilistic altern…

As mentioned in the blog post, clusters allow horizontal scalability and daisy chaining, so you can allocate more memory for your views even if you run up against the limits of how much memory you can fit on a single machine. We've got plans in the works to support out-of-core execution, too.

> Also they do not integrate at all with custom data types in Postgres IME. E.g. an enumeration in your table will mean materialize can’t read the table as a source. Lame.

We're aware of this and are working on a fix. There are two tracking issues, if you'd like to follow along:

* #6818 (https://github.com/MaterializeInc/materialize/issues/6818) is specifically about supporting PostgreSQL enum types * #15073 (https://github.com/MaterializeInc/materialize/issues/15073) is about handling PostgreSQL types that are unknown to Materialize in a more general purpose way *

Re: The next generation of Materialize

#32

Post by andrenotgiant » Mon, Oct 03, 2022, 7:21 PM UTC

cube2222 wrote: post #11

This is really impressive! I've been following Materialize as their blog posts are a great source of inspiration when working on OctoSQL[0] (a CLI SQL dataflow engine), but was a bit surprised with how few data sources they were supporting (basically Kafka and Postgres based on their docs), but now that they're switching/pivoting to being a database themselves, this makes much more sense. I also think the architectur…

Thanks! To answer your question:

The core components of Materialize have always been licensed under the BSL [0], with a conversion to the Apache v2.0 License four years from the date of publication. These components (now called the storage, compute, and adapter layers) continue to be developed in the open under the same license, but are no longer packaged or supported for on-premise deployment.

Timely Dataflow and Differential Dataflow [1] are the open source frameworks that have always been at the heart of the compute layer in Materialize.

If you're referencing the downloadable binary when you mentioned "the open-source version", we decided the opportunity cost of splitting our attention and continuing to release updated binary versions by "rebundling" Materialize is too high. We have a small team and ambitious plans for Materialize, (maybe some `WITH RECURSIVE`?) so we don't plan to release updates to the binary.

[0] https://github.com/MaterializeInc/materialize/blob/main/LICE... [1] https://timelydataflow.github.io/differential-dataflow/intro...

Re: The next generation of Materialize

#33

Post by dagss » Mon, Oct 03, 2022, 8:02 PM UTC

mozinator wrote: post #15

The poor man's version of Materialize that I implemented is the following: Step 1) Find all "paths" between tables * Use the postgres information schema to get all relations * Use npm library graph-cycles to see if there are any graph cycles. If so.. some relations go on a blacklist. * Use npm library topopsort to sort the graph * Traverse the graph and find all possible paths from and to tables * Generate SQL querie…

We do something similar, but in 2), instead of using the outbox pattern, we make use (in several different settings) of integers that are guaranteed to increment in commit order, then each consumer can track where their cursor is on the feed of changes. This requires some more care to get that sequence number generated in a safe way, but it means that publishers of changes don't need one outbox per consumer or similar.

Then you can have "processes" that query for new data in an input table, and update aggregates/derived tables from that simply by "select * ... where ChangeSequenceNumber > @MaxSequenceNumberFromPreviousExecution"...

The idea here implemented for Microsoft SQL for the OLTP case:

https://github.com/vippsas/mssql-changefeed https://github.com/vippsas/mssql-changefeed/blob/main/MOTIVA...

If you are ingesting events from e.g. Kafka or other similar sources it is easier to assign sequence numbers though and this is not needed

Re: The next generation of Materialize

#34

Post by ivanvas » Mon, Oct 03, 2022, 9:12 PM UTC

There seems to be an overlap with ksqlDB functionality. ksqlDB computation part is based on Kafka Streams. https://github.com/confluentinc/ksql

Re: The next generation of Materialize

#35

Post by School-Cotton » Mon, Oct 03, 2022, 9:30 PM UTC

chekibreki wrote: post #13

Can someone please explain, without technical buzzwords, what Materialize is? I checked the docs but they mention streaming DB, Timely and Differential Dataflow which I don't know either.

I have worked at Materialize since 2019. The elevator pitch is that it is a database that lets you maintain the results of queries in memory (and now S3) and have them update in real time so the current result is always available.

Re: The next generation of Materialize

#36

Post by jonstewart » Mon, Oct 03, 2022, 9:31 PM UTC

I’m starting to play with differential-dataflow in a new Rust project. On the one hand, it’s cool. On the other, it could use better documentation.

With Materialize the database, it really depends on price/performance whether I could use it (could it be really cheap when idle, like Aurora Serverless?), but two things that would make it easier to use would be the ability to purchase it through AWS Marketplace and to deploy it/use it in my organization’s own AWS account. As an enterprise dev team senior manager I then do not have to go through a vendor approval process or deal with my procurement department, nor do I need to worry about third party data control.

Re: The next generation of Materialize

#37

Post by School-Cotton » Mon, Oct 03, 2022, 9:35 PM UTC

TOMDM wrote: post #7

edude03 wrote: post #2

The downside is they pivoted away from supporting running it yourself. The technology is certainly exciting but also changes the target demographic. I'm curious to see how it plays out

Such a shame, I had an ideal use case for a product I'm working on, but there's no way I'm shackling it to cloud vendors. Just not a good fit for this product. I really hope materialize eventually inspires something like it in the open source world.

Materialize itself is under a BSL license that makes every commit automatically convert to open source four years after it lands.

Re: The next generation of Materialize

#38

Post by cryptonector » Mon, Oct 03, 2022, 11:18 PM UTC

I use PG with an alternative materialized views implementation[0] that is pure PlPgSQL and that exposes real tables that can be used to write to in triggers, and where the views can be marked stale too.

This means hand-coding triggers to keep the materializations up to date, or else to mark them as out of date (because maybe some operations would be slow or hard to hand-code triggers for), but this works remarkably well.

As a bonus, I get an update history table that can be used to generate updates to external systems.

In principle one can get the AST for a VIEW's query from the PG catalog and use that generate triggers on the tables it queries to keep it up to date. In practice that's only trivial for some kinds of queries, and I've not written such a tool yet.

[0] https://github.com/twosigma/postgresql-contrib/blob/master/m...

Re: The next generation of Materialize

#39

Post by SamuelHu » Tue, Oct 04, 2022, 1:18 AM UTC

TOMDM wrote: post #7

edude03 wrote: post #2

The downside is they pivoted away from supporting running it yourself. The technology is certainly exciting but also changes the target demographic. I'm curious to see how it plays out

Such a shame, I had an ideal use case for a product I'm working on, but there's no way I'm shackling it to cloud vendors. Just not a good fit for this product. I really hope materialize eventually inspires something like it in the open source world.

Please also take a look at https://github.com/risingwavelabs/risingwave if you are looking for advanced streaming databases. It is under Apache License and also support on-prem deployment (docker, kubernetes) with full function set of distributed clustering, compute-storage disaggregation, etc.. Let me know if anything.

Re: The next generation of Materialize

#40

Post by ibgeek » Tue, Oct 04, 2022, 3:48 AM UTC

School-Cotton wrote: post #35

chekibreki wrote: post #13

Can someone please explain, without technical buzzwords, what Materialize is? I checked the docs but they mention streaming DB, Timely and Differential Dataflow which I don't know either.

I have worked at Materialize since 2019. The elevator pitch is that it is a database that lets you maintain the results of queries in memory (and now S3) and have them update in real time so the current result is always available.

Can you describe some use cases? The only one that comes to mind is powering dashboards.

That said, do you support persistent tables or time travel? (E.g., accessing the contents of the view as it was at time t). If not, how should a situation where multiple independent clients read the view be handled? If two people load the same dashboard but see differences based on when the table was read, that might cause confusion.