Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#81

Post by solidangle »

solidangle wrote: post #32

> Blazing fast results I highly doubt this, given that the query engine is interpreted and non-vectorized. Queries are 10x to a 100x slower on a simple query, and 100x to 1000x slower on a query with large aggregations and joins without compilation of vectorization. > Full SQL Exploration Except for window functions it seems. These actually matter to data analysts.

FWIW, I do think this project is really cool. I should have taken a little bit more time to write this comment, as it's overly negative right now.

Re: Materialize: A Streaming Data Warehouse

#82

Post by drej »

I really like the pg protocol (like e.g. Cockroach), it let me use my usual tools. There are a few things I noticed:

1. It has a fairly rich support for types - these new-ish SQL engines often lack quite a lot of things, but this seems pretty decent. 2. I don't see any comparisons to KSQL, which seems to be the primary competitor. 3. Read the license. Read it carefully. It has a weird "will become open source in four years" clause, so keep that in mind. It also disallows it being hosted for clients to use (esentially as a DBaaS).

Re: Materialize: A Streaming Data Warehouse

#83

Post by taurath »

benesch wrote: post #30
slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

> I just want to say this is a very dangerous assumption to make. I think we're actually arguing the same points here. It's not that every use case needs single-digit millisecond latencies! There are plenty of use cases that are satisfied by batch jobs running every hour or every night. But when you do need real-time processing, the current infrastructure is insufficient. When you do need single-digit latency, runnin…

I could see a usecase for rapid automated A/B testing, using ML to react to performance metrics. Why have human editors on cnn.com when you can in real time do story selection based on view traffic?

That said, hope we as technologists can find some better use cases than just stealing more of people’s a attention.

Re: Materialize: A Streaming Data Warehouse

#87

Post by choppaface »

benesch wrote: post #51
yomly wrote: post #40

Earlier quoted context omitted.

This reminds me a lot about Noria DB. Wonder if anyone familiar with both can shed any further light?

Indeed, Materialize is quite similar to Noria, and has the Frank McSherry stamp of awesomeness. [0] We know many of the Noria folks and have a lot of respect for them and their work. I also worked on the Noria project for a summer in college, and am a full-time engineer at Materialize now. The biggest difference is one of intended use. Noria is, first and foremost, a research prototype, intended to explore new ideas…

Do you have an example of where one might use WITH RECURSIVE?

Re: Materialize: A Streaming Data Warehouse

#88

Post by chris_overseas »

wiradikusuma wrote: post #84

Materialize connects directly to event stream processors (like Kafka) --- how about Pulsar? (Goggling doesn't yield anything useful, Materialize and Pulsar are both name of brands of other things)

Connecting Pulsar and Materialize is of interest to me too and something I might try when I find time to do so. Note that Pulsar does have a Kafka compatibility layer already[0], so it might just work out of the box. If you try this I'd be keen to hear how it goes.

[0] https://pulsar.apache.org/docs/en/adaptors-kafka/

EDIT: I don't think this adaptor will work after all, it works by replacing the Kafka Java client library with its own, so is only applicable to Java applications.

Re: Materialize: A Streaming Data Warehouse

#89

Post by thom »

jacques_chester wrote: post #50
benesch wrote: post #45

Earlier quoted context omitted.

I agree wholeheartedly with your take! Window functions are a particular favorite of mine, but we haven’t seen much customer demand for them yet, so they haven’t been officially scheduled on the roadmap. They require some finesse to support in a streaming system, as you have to reconstruct the potentially large window whenever you receive new data. Probably some interesting research to be done here, or at least some…

I have a strong suspicion that bitemporalism makes a lot of these problems less problematic. The actual volumes of data are the same, but the all-or-nothingness of windowing over very large data sets in order to avoid missing anything that arrived late goes away. I wrote shambolic stream-of-consciousness notes on it several years ago: https://docs.google.com/document/d/1ZlPp099_fV1lyYWACSyuWY_j... The gist being that…

This feels like the philosophical conclusion that Kafka Streams has made, i.e. you don't have a strict watermark, and if you really want you can theoretically keep updating and retracting data forever, and build a pipeline that magically stays in sync.

Re: Materialize: A Streaming Data Warehouse

#90

Post by jamii »

derefr wrote: post #16
slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

> Secondly, almost all data is useless in its raw form. The analysts had to perform ELT jobs on their data in the warehouse to clean, dedupe, aggregate, and project their business rules on that data. These functions often require the database to scan over historical data to produce the new materializations of that data. The point of Materialize, from my understanding, is that you don't put things into the data wareho…

> Instead, you register persistent, stateful enrichment "streaming jobs" (i.e. incrementally-materialized views) into the data warehouse; and then, when data comes into a table upstream of these views, it gets streamed into and through the related job to incrementally populate the matview.

This is correct. There is an example in the docs of dumping json records into a source and then using materialized views to normalize and query them - https://materialize.io/docs/demos/microservice/

Post reply on HN