Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#71

Post by manigandham »

1290cc wrote: post #47

Pretty cool tech although I feel they may have missed the moment as AWS, Azure and GCP are becoming hypercompetitive to solve all things related to data/storage. Azure has been churning out major updates to its services and clearly taking inspiration from companies like Snowflake. AWS I think hesitated to compete with Snowflake as they were running on AWS anyway - win/win for them. Snowflake had incredible timing as…

Google's data product suite seems to be the most advanced today with a completely no-ops approach and very solid primitives to work with.

Snowflake still doesn't an answer to streaming data (other than their fragile Kafka connector) and Azure Synapse still isn't publically available even months after their announcement.

AWS had a good headstart but they keep piling on more products and features that it's now a big mess and requires yet another tool (AWS Lake Formation) just to wire it all up.

Re: Materialize: A Streaming Data Warehouse

#72

Post by manigandham »

chrisjc wrote: post #42
programmarchy wrote: post #17

Isn't BigQuery real-time as well? For me, the wow factor is that you can host this yourself.

I'm unfamiliar with BQ, but from what I understand BQ doesn't have any concept of streams... with maybe an exception for change-streams (change tracking) and that's very different and less powerful than what Materialize is able to do. After all, in BQ you're only able to query what has been already has been ingested or is being digested and that is somewhat by definition not 'real-time'.

GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam).

There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job.

https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...

Re: Materialize: A Streaming Data Warehouse

#73

Post by manigandham »

programmarchy wrote: post #17

Isn't BigQuery real-time as well? For me, the wow factor is that you can host this yourself.

BigQuery does have streaming ingest and can pickup new rows immediately but you still have to run a query to get the latest results.

This system is more like BigQuery with materialized views that are constantly updated as new data arrives.

Re: Materialize: A Streaming Data Warehouse

#74

Post by wrmsr »

vhold wrote: post #62
justlexi93 wrote: post #57

Clickhouse has materilized views and is free.

I think the biggest difference is that Materialize can do any kind of SQL join on many tables at once. Clickhouse materialized views can only reference one table. What I'd like to know is if that would enable basically implementing social networks as just 3 tables and one materialized view, and how it would scale and perform. Users, Relationships, Post, and a Feed materialized view that simply joins them together wit…

In the real world it is inevitably a lot more than just those 3 tables - add 'groups', different types of groups, different privacy settings, different per-user feed preferences, experiments, and any number of other things which _can_ be expressed as pure, normalized, joined tables in a matview but make naive approaches a lot less likely to actually work in prod.

In my experience the most successful approach to this is a midpoint - you materialize/denormalize enough to feed your app endpoints and search engines but retain flexibility in searching those fat but instantly available docs, and relatedly you also don't always need to preemptively materialize absolutely everything in any particular view - see https://engineering.fb.com/data-infrastructure/dragon-a-dist... . Without being able to transparently operate on arbitrarily partially populated matviews you are locked into a self-defeating all-or-nothing system that is likely to culturally do more harm than good with its rigidity. Imagine for example if there were no 'caches', just a binary choice of precomputing everything ahead of time or recomputing everything every time. Neither extreme is sufficient for all cases and real applications are comprised of many different points on that spectrum.

Re: Materialize: A Streaming Data Warehouse

#75

Post by manigandham »

Congrats on the launch, always nice to see new products.

This is an interesting mix between the (now obsolete) PipelineDB, TimescaleDB with continuous aggregates, Kafka and other message systems with KSQL/ksqlDB/KarelDB, stream processing engines like Spark, and typical RDBMS like SQL Server with materialized views.

The amount of research to support complex and layered queries definitely sets this apart.

Re: Materialize: A Streaming Data Warehouse

#76

Post by grumblestumble »

streblo wrote: post #36

How does materialize compare in performance (especially ingress/egress latency) to other OLAP systems like Druid or ClickHouse? Would love to see some benchmarks.

This appears to not be fully open source: https://github.com/MaterializeInc/materialize/blob/master/LI...

Re: Materialize: A Streaming Data Warehouse

#78

Post by jerglingu »

mason55 wrote: post #41

Not sure how the featuresets compare but AWS is releasing materialized views for Redshift sometime soon and one of the things it will support is incremental refresh (assuming your view meets some criteria). I'm sure Materialize is better at this since it's purpose-built but if you're on Redshift you can get at least some of the benefits of incremental materialize.

Materialized views, the upcoming ability to query directly from an RDS transactional DB through Redshift, and finally true separation of compute vs storage with RA nodes IMO make Redshift the market leader by a huge margin now. I haven't actually tested RA nodes yet, but if performance is even just a fraction of legacy nodes then the competition is already dead. Redshift already is one of the best engines to optimize a query

Re: Materialize: A Streaming Data Warehouse

#79

Post by vhold »

wrmsr wrote: post #74
vhold wrote: post #62

Earlier quoted context omitted.

I think the biggest difference is that Materialize can do any kind of SQL join on many tables at once. Clickhouse materialized views can only reference one table. What I'd like to know is if that would enable basically implementing social networks as just 3 tables and one materialized view, and how it would scale and perform. Users, Relationships, Post, and a Feed materialized view that simply joins them together wit…

In the real world it is inevitably a lot more than just those 3 tables - add 'groups', different types of groups, different privacy settings, different per-user feed preferences, experiments, and any number of other things which _can_ be expressed as pure, normalized, joined tables in a matview but make naive approaches a lot less likely to actually work in prod. In my experience the most successful approach to this…

That's fair. It will be interesting to see what people do along those lines, creating various materialized views, joining them at query time, chaining materialized views, and I think most important to your point, creating new kinds of sinks for the updates.

Right now Kafka is the only sink ( https://materialize.io/docs/sql/create-sink/ ), but because it integrates with Confluent's schema registry, I'm guessing it should work well with many of Confluent Connect's sinks, ( https://docs.confluent.io/current/connect/managing/connector... ). Elasticsearch would be an especially useful sink connector I think to your point.

I haven't used any of these things together, so right now I'm totally speculating on the potential.

What I'm mostly envisioning is that there are a lot of smaller scale applications where the complexity of adding an activity feed just isn't worth it. But if you could implement a feature like that trivially it could be game changing.

Re: Materialize: A Streaming Data Warehouse

#80

Post by Ar-Curunir »

benesch wrote: post #49
Ar-Curunir wrote: post #43

Earlier quoted context omitted.

That’s built on top of differential data flow, the same thing underlying Materialize

It’s not, actually. Noria has its own custom dataflow engine.

oops, for some reason I thought that the paper talked about using differential dataflow, but it seems that I was mistaken.
Post reply on HN