Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Materialize: A Streaming Data Warehouse

materialize.io

Post reply on HN
Print view View original

31–40 of 103 posts

Re: Materialize: A Streaming Data Warehouse

#31

Post by benesch »

jacques_chester wrote: post #28
benesch wrote: post #25

Earlier quoted context omitted.

Yes, absolutely. Entire dissertations have been written on the topic: https://dash.harvard.edu/bitstream/handle/1/14226097/KATE-DI... Efficiently maintaining views over arbitrarily complex computations is one of the two hard problems in computer science for a reason [0]. [0]: https://martinfowler.com/bliki/TwoHardThings.html

Never let HN haters (among whom I am frequently numbered, it ought to be noted) get you down. After skimming a bit of the differential dataflow writing I am really impressed. This is deep computer science doing what it does best, which is to do much more with much less.

I’m glad to hear it! One of my favorite ways to view Materialize is as bringing differential dataflow to the masses. Differential dataflow is deep, elegant stuff, but it requires some serious CS chops to grok.

SQL is lacking in elegance but abundant in popularity. Building the SQL translation layer has been a fun exercise in bridging the two worlds.

Re: Materialize: A Streaming Data Warehouse

#32

Post by solidangle »

> Blazing fast results

I highly doubt this, given that the query engine is interpreted and non-vectorized. Queries are 10x to a 100x slower on a simple query, and 100x to 1000x slower on a query with large aggregations and joins without compilation of vectorization.

> Full SQL Exploration

Except for window functions it seems. These actually matter to data analysts.

Re: Materialize: A Streaming Data Warehouse

#33

Post by benesch »

solidangle wrote: post #32

> Blazing fast results I highly doubt this, given that the query engine is interpreted and non-vectorized. Queries are 10x to a 100x slower on a simple query, and 100x to 1000x slower on a query with large aggregations and joins without compilation of vectorization. > Full SQL Exploration Except for window functions it seems. These actually matter to data analysts.

Considerations are completely different in a streaming context. It’s not so much about how fast you can churn through terabytes of data; it’s more about how quickly you can turn around the incremental computation with each new datum. There’s some serious research behind this product, in timely and differential dataflow, and I’d encourage you to check out some of that research before making sweeping performance claims. Frank’s blog post on TPC-H is a good place to start: https://github.com/frankmcsherry/blog/blob/master/posts/2017...

We definitely have some performance engineering work to do in Materialize, but don’t let the lack of vectorization scare you off. It’s just not as important for a streaming engine.

Re: Materialize: A Streaming Data Warehouse

#34

Post by edmundsauto »

solidangle wrote: post #32

> Blazing fast results I highly doubt this, given that the query engine is interpreted and non-vectorized. Queries are 10x to a 100x slower on a simple query, and 100x to 1000x slower on a query with large aggregations and joins without compilation of vectorization. > Full SQL Exploration Except for window functions it seems. These actually matter to data analysts.

It's one thing to be skeptical and ask for evidence of speed, another to dismiss them out of hand due to a casual review of their website. Or did I miss it that you tried it out and found it wanting?

I work in an org with > 100 data scientists. I bet that 50% have never used window functions. I would guess than fewer than 20% know how to write one.

Your intent in this comment is unclear, but if you were looking to provide actionable feedback, you might want to reconsider your tone. This project looks like a pretty impressive feat of applied CS theory to doing useful stuff.

Re: Materialize: A Streaming Data Warehouse

#38

Post by benesch »

gbrits wrote: post #37

Is this similar to TimescaleDB's Continuous Aggregates? Interested in knowing the overlap / differences.

Similar in concept, but much more powerful in execution. We can incrementally materialize practically any SQL 92 query, with the killer features being joins and correlated subqueries. I’m not super familiar, but TimescaleDB’s continuous aggregates (just on a cursory glance) don’t support stacked aggregates, for example: https://github.com/timescale/timescaledb/issues/1400

Re: Materialize: A Streaming Data Warehouse

#39

Post by chrisjc »

benesch wrote: post #18
chrisjc wrote: post #9

Would it be fair to say this is a more OLAP-oriented approach to what KSqlDB (not KSql, but https://ksqldb.io/ ) does? Seems that it's perhaps lacking the richness of how ksqldb uses Kafka Connectors (sinks and sources), but I don't see any reason you couldn't use Materialize in conjunction with ksqldb. Eg: KC-source --> ksql --> materialize --> kafka --> KC-sink Question to Materialize... What connectors (sinks and…

> Would it be fair to say this is a more OLAP-oriented approach to what KSqlDB (not KSql, but https://ksqldb.io/ ) does? I'm not sure I'd say it's "more OLAP." ksqlDB is about as OLAP as it gets, considering it doesn't support any sort of transactions or consistency. We think Materialize is quite a bit more powerful than what ksqlDB offers, thanks to the underlying technologies (timely/differential). For example, our…

Thanks for the detailed response. At this point i think the onus is on me to go and take a deeper look into timely/differential.

Important "sources" to me are obviously Kafka, but also MySQL and Mongo. Important "sinks" would be Snowflake (maybe through S3, or directly though PUT) and ElasticSearch. Although I imagine you might soon be telling me that you don't need a data warehouse once you have Materialize :)

Re: Materialize: A Streaming Data Warehouse

#40

Post by yomly »

luhn wrote: post #12

I didn't really understand what the product actually did after reading this blog post or the products page. I found the docs much more edifying: > Materialize lets you ask questions about your data, and then get the answers in real time. > Why not just use your database’s built-in functionality to perform these same computations? Because your database often acts as if it’s never been asked that question before, which…

This reminds me a lot about Noria DB. Wonder if anyone familiar with both can shed any further light?
Post reply on HN