Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

61–70 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#61

Post by mason55 »

chrisjc wrote: post #48
mason55 wrote: post #41

Not sure how the featuresets compare but AWS is releasing materialized views for Redshift sometime soon and one of the things it will support is incremental refresh (assuming your view meets some criteria). I'm sure Materialize is better at this since it's purpose-built but if you're on Redshift you can get at least some of the benefits of incremental materialize.

It's been a while since I've used Redshift, but isn't it still dependent on data coming in via a COPY from S3? Any sort of Redshift materialized view offering would depend on batches of data landing in an underlying table or tables. The closest service offering from AWS is probably using Kinesis analytics (or Flink on KA) using their flavor of streaming SQL to join Kinesis streams forming new ones.

With the introduction of Spectrum you can back a Redshift table with data in S3 directly. I'm not sure how that interacts with the materialized views though. Probably not supported yet but I would expect it to be eventually.

Re: Materialize: A Streaming Data Warehouse

#62

Post by vhold »

justlexi93 wrote: post #57

Clickhouse has materilized views and is free.

I think the biggest difference is that Materialize can do any kind of SQL join on many tables at once. Clickhouse materialized views can only reference one table.

What I'd like to know is if that would enable basically implementing social networks as just 3 tables and one materialized view, and how it would scale and perform.

Users, Relationships, Post, and a Feed materialized view that simply joins them together with an index of user_id and post_timestamp.

As relationships and messages are created or deleted, the feed view is nearly instantly updated. The whole entire view service logic then is just one really fast query. "select user,post,post_timestamp from feed where user_id = current_user and post_timestamp <= last_page_post_timestamp order by post_timestamp desc limit page_size"

Re: Materialize: A Streaming Data Warehouse

#63

Post by benesch »

gaogao wrote: post #56

The "you may not cluster any server instances of the Licensed Work together for one use" in the license is a fairly tricky clause. Under this clause, how would one run a fault-tolerant instance of Materialize?

You would need to contact us to buy an enterprise license. (Or wait four years for the license to convert to Apache 2.)

Re: Materialize: A Streaming Data Warehouse

#65

Post by vhold »

gbrits wrote: post #46

Looks promising! Can materialized views be backfilled?

According to the documentation at https://materialize.io/docs/overview/architecture/

> Streaming sources must receive all of their data from the stream itself; there is no way to “seed” a streaming source with static data. However, you can union streaming and file sources in views, which accomplishes a similar outcome.

What I find odd is that the documentation doesn't show UNION support in the SELECT documentation : https://materialize.io/docs/sql/select/

Perhaps just an oversight in the documentation?

Re: Materialize: A Streaming Data Warehouse

#66

Post by benesch »

jacques_chester wrote: post #60
benesch wrote: post #54

Earlier quoted context omitted.

We’re thinking along very similar lines! We’ve got some of our thoughts around evolving how timestamps work in Materialize written down here: https://github.com/MaterializeInc/materialize/issues/1309

Sort of -- the problem I see in the event time / processing time distinction is that it's about instants rather than intervals . There are a number of models and queries that are not reliably expressible with instants alone, unless you reinvent intervals with them. For example, if I rely on "updated-at" and infer that whatever record has the latest updated-at is the "current" record, then I may create the illusion th…

> For example, if I rely on "updated-at" and infer that whatever record has the latest updated-at is the "current" record, then I may create the illusion that there are no gaps in my facts. That may not be so.

I believe that notion is captured by timely's capabilities [0]. Your capability has a current time, and you can only produce records at or greater than the current time. So you could produce a record at, say, t + 3, then t + 5, and then produce a record at t + 1. But not until you downgrade your capability to t + 6 will the record at t + 5 be considered final; downgrading your capability is how you indicate that you have the correct and final set of facts for all times less than t.

If your events can arrive out of order forever, then you have a problem, as you'll never be able to downgrade your capability because you'll never be willing to mark a time as "final." That's where bitemporalism (as mentioned in that issue I linked previously) comes into play. You can mark a result as final as of some processing time, and then issue corrections as of some processing time in the future if some out-of-order data arrives. Materialize will (likely) gain support for bitemporalism eventually, and the underlying dataflow engine supports arbitrary-dimension timestamps already.

Would be happy to chat about this more, if you're curious, but I feel like this discussion is getting a bit unwieldy for an HN thread! (At the very least I might need to put you in touch with Frank.) Feel free to reach out on GitHub [1] or our Gitter [2], or shoot me an email at benesch@materialize.io.

[0]: https://docs.rs/timely/0.11.1/timely/dataflow/operators/stru...

[1]: https://github.com/MaterializeInc/materialize/issues

[2]: https://gitter.im/MaterializeInc/community

Re: Materialize: A Streaming Data Warehouse

#67

Post by frankmcsherry »

jacques_chester wrote: post #60
benesch wrote: post #54

Earlier quoted context omitted.

We’re thinking along very similar lines! We’ve got some of our thoughts around evolving how timestamps work in Materialize written down here: https://github.com/MaterializeInc/materialize/issues/1309

Sort of -- the problem I see in the event time / processing time distinction is that it's about instants rather than intervals . There are a number of models and queries that are not reliably expressible with instants alone, unless you reinvent intervals with them. For example, if I rely on "updated-at" and infer that whatever record has the latest updated-at is the "current" record, then I may create the illusion th…

The underlying compute framework, differential dataflow, supports multi-temporal timestamps. The Crux folks were at one point looking at it for Juxt, though not sure what they concluded.

Re: Materialize: A Streaming Data Warehouse

#68

Post by benesch »

vhold wrote: post #65
gbrits wrote: post #46

Looks promising! Can materialized views be backfilled?

According to the documentation at https://materialize.io/docs/overview/architecture/ > Streaming sources must receive all of their data from the stream itself; there is no way to “seed” a streaming source with static data. However, you can union streaming and file sources in views, which accomplishes a similar outcome. What I find odd is that the documentation doesn't show UNION support in the SELECT documentation :…

Good catch, thanks! I've filed an issue about this: https://github.com/MaterializeInc/materialize/issues/2056

Though it's not yet documented, we do support all of the SQL set operations [0]: UNION [ALL], INTERSECT [ALL], and EXCEPT [ALL].

[0]: https://www.postgresql.org/docs/current/queries-union.html

Re: Materialize: A Streaming Data Warehouse

#70

Post by kiwicopple »

For anyone that might be considering trying something similar with their own Postgres database (PG10+), we recently opensourced this: https://github.com/supabase/realtime

It's an Elixir (Phoenix) server that listens to PostgreSQL's native replication, transforms it into JSON, then blasts it over websockets.

I see that Materialize are using Debezium, which will give you a similar result, just with connectors to Kafka etc

Post reply on HN