Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

91–100 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#91

Post by jamii »

drej wrote: post #82

I really like the pg protocol (like e.g. Cockroach), it let me use my usual tools. There are a few things I noticed: 1. It has a fairly rich support for types - these new-ish SQL engines often lack quite a lot of things, but this seems pretty decent. 2. I don't see any comparisons to KSQL, which seems to be the primary competitor. 3. Read the license. Read it carefully. It has a weird "will become open source in four…

The main difference vs KSQL is that we support standard SQL (roughly SQL92 + correlated subqueries + json so far) and provide strong consistency:

KSQL has a distinction between streams and tables, effectively giving you control over how views are materialized but also forcing you to do that work yourself. In Materialize you just write the same SQL that you would for a batch job and the planner figures out how to transform it into a streaming dataflow.

KSQL is also eventually consistent - late arriving records might cause changes to previously emitted answers. Materialize won't give you an answer for a given timestamp until it can guarantee that the answer is correct and won't be retroactively updated later.

Expect to see some performance comparisons soon too.

Re: Materialize: A Streaming Data Warehouse

#92

Post by jamii »

chris_overseas wrote: post #88
wiradikusuma wrote: post #84

Materialize connects directly to event stream processors (like Kafka) --- how about Pulsar? (Goggling doesn't yield anything useful, Materialize and Pulsar are both name of brands of other things)

Connecting Pulsar and Materialize is of interest to me too and something I might try when I find time to do so. Note that Pulsar does have a Kafka compatibility layer already[0], so it might just work out of the box. If you try this I'd be keen to hear how it goes. [0] https://pulsar.apache.org/docs/en/adaptors-kafka/ EDIT: I don't think this adaptor will work after all, it works by replacing the Kafka Java client li…

Adding sources is pretty easy. We just need them to provide:

* timestamped changes eg "record foo was deleted at time 42"

* watermarks eg "I've now sent you all the changes up to time 42"

Here's the implementation of the kafka source - https://github.com/MaterializeInc/materialize/blob/master/sr.... (There's also a bit of code in the parser/planner to wire up the CREATE SOURCE syntax, but it's fairly trivial.)

Re: Materialize: A Streaming Data Warehouse

#93

Post by thom »

slap_shot wrote: post #11

> We believe that streaming architectures are the only ones that can produce this ideal data infrastructure. I just want to say this is a very dangerous assumption to make. I run a company that helps our customers consolidate and transform data from virtually anywhere in their data warehouses. When we first started, the engineer in me made the same declaration, and I worked to get data into warehouses seconds after a…

I work in the sports stats industry and we've been talking to Materialize for a while. We have a lot of demand for real time data, especially from the media and gambling sectors. The SLAs are mostly in seconds not milliseconds, so we currently don't have to wring every last ounce of latency out of our pipeline, but I'm very excited about the product because it offers a good balance of performance and developer productivity. I can even imagine a universe in which we directly offer the capability to customers to be able to build their own real-time KPIs in a language that is fairly accessible and easy to hire for.

Re: Materialize: A Streaming Data Warehouse

#94

Post by jamii »

streblo wrote: post #36

How does materialize compare in performance (especially ingress/egress latency) to other OLAP systems like Druid or ClickHouse? Would love to see some benchmarks.

We're working on benchmarks, but I expect that at the moment Materialize will be slower for one-off queries but faster for frequently repeated queries.

Re: Materialize: A Streaming Data Warehouse

#95

Post by thom »

kiwicopple wrote: post #70

For anyone that might be considering trying something similar with their own Postgres database (PG10+), we recently opensourced this: https://github.com/supabase/realtime It's an Elixir (Phoenix) server that listens to PostgreSQL's native replication, transforms it into JSON, then blasts it over websockets. I see that Materialize are using Debezium, which will give you a similar result, just with connectors to Kafka…

The magic here isn't that it's giving you streaming updates of a database, it's that it's making some of those updates ridiculously fast. The point is that if you have a materialized view that takes a long time to update (or just a stack of views that are slow to return results), you now have results instantly whenever changes come in. If you built a bunch of code to work around this and do clever invalidation of cached data, you get to throw all that away. The input/output into the system is less of an issue overall.

Re: Materialize: A Streaming Data Warehouse

#96

Post by jacques_chester »

thom wrote: post #89
jacques_chester wrote: post #50

Earlier quoted context omitted.

I have a strong suspicion that bitemporalism makes a lot of these problems less problematic. The actual volumes of data are the same, but the all-or-nothingness of windowing over very large data sets in order to avoid missing anything that arrived late goes away. I wrote shambolic stream-of-consciousness notes on it several years ago: https://docs.google.com/document/d/1ZlPp099_fV1lyYWACSyuWY_j... The gist being that…

This feels like the philosophical conclusion that Kafka Streams has made, i.e. you don't have a strict watermark, and if you really want you can theoretically keep updating and retracting data forever, and build a pipeline that magically stays in sync.

Partially, in my understanding, but not fully. An advantage of bitemporalism that is hard to recreate is queries about past and future states of belief. "What do I believe is true today?" works well with accumulation and reaction and with standard normalised schemata.

"What do I believe I believed yesterday?" is slightly harder and needs additional information to be stored. You can rewind a stream and replay it up to the point of interest, but that can be quite slow.

"What did I believe today would be, last week?", "What is the history of my belief about X?", "have I ever believed Y about X?" etc are much harder to answer quickly without full bitemporalism. So too the problem of having implicit intervals that are untrue, which is where "updated at" can be so misleading.

Re: Materialize: A Streaming Data Warehouse

#97

Post by yayr »

I am curious about the physical storage. Is it purely in-memory or is there a disk persistency possible? Is there some kind of data compression applied or what are the memory needs of it? Is it a row or column based data persistence pattern?

Re: Materialize: A Streaming Data Warehouse

#98

Post by chrisjc »

manigandham wrote: post #72
chrisjc wrote: post #42

Earlier quoted context omitted.

I'm unfamiliar with BQ, but from what I understand BQ doesn't have any concept of streams... with maybe an exception for change-streams (change tracking) and that's very different and less powerful than what Materialize is able to do. After all, in BQ you're only able to query what has been already has been ingested or is being digested and that is somewhat by definition not 'real-time'.

GCP has PubSub which is their streaming/messagebus product. Cloud Dataflow is their stream/batch processing engine (backed by Apache Beam). There's a new feature called "Dataflow SQL" that lets you submit a declarative SQL query through the BQ UI as a job to Dataflow which can join both BigQuery tables and PubSub topics in the same job. https://cloud.google.com/dataflow/docs/guides/sql/dataflow-s...

Thank you for the correction. Sounds like a very slick solution bringing batch and streaming (the B and EAM obviously) together.

Re: Materialize: A Streaming Data Warehouse

#99

Post by jamii »

polskibus wrote: post #77

Looking back at the project, knowing what you know now, if you were to start again (but without obtained rust skills), would you go with rust again or pick another toolbox?

> would you go with rust again

Absolutely. Even aside from safety, rust has so many quality of life improvements over C++ - algebraic data types, pattern matching, checked errors instead of exceptions, sane standard library, easy packaging, readable error messages, #[must_use], immutable utf8 strings, traits instead of templates, hygienic macros etc.

Other than compile times and editor tooling, which are being actively worked on, the only real pain point I can think of is the poor support for custom allocators.

As for other alternatives, I don't personally have strong opinions on go but I think it's notable that none of the ex-cockroachdb folks at materialize suggested using it instead of rust.

Re: Materialize: A Streaming Data Warehouse

#100

Post by jadbox »

yomly wrote: post #40
luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

This reminds me a lot about Noria DB. Wonder if anyone familiar with both can shed any further light?

It also sounds a bit like InfluxDB to me..
Post reply on HN