Materialize: A Streaming Data Warehouse
materialize.io
Materialize: A Streaming Data Warehouse
1–10 of 103 posts
Re: Materialize: A Streaming Data Warehouse
#2Re: Materialize: A Streaming Data Warehouse
#3I'm not sure if it's a mistake or just some cheeky humor but the "Known Limitations" link on their Docs page returns a 404.
Re: Materialize: A Streaming Data Warehouse
#4I'm not sure if it's a mistake or just some cheeky humor but the "Known Limitations" link on their Docs page returns a 404.
It looks like a caching bug somewhere in the web pipeline. It isn't running on Materialize, sadly.
There are a pile of known limitations, and our goal is to be 100% clear about them.
Re: Materialize: A Streaming Data Warehouse
#5I'm not sure if it's a mistake or just some cheeky humor but the "Known Limitations" link on their Docs page returns a 404.
It looks like a caching bug somewhere in the web pipeline. It isn't running on Materialize, sadly. There are a pile of known limitations, and our goal is to be 100% clear about them.
Hi Frank! Congrats on the progress - I remember listening to you describing Materialized at SIGMOD'19 in the big hall - it sounded like magic back then, glad to see it come true :) I have a question now - can you subscribe to view updates somehow, instead of querying them on an ad-hoc basis?
Re: Materialize: A Streaming Data Warehouse
#6(Linked in the post but) github repo: https://github.com/MaterializeInc/materialize
Re: Materialize: A Streaming Data Warehouse
#7See also https://news.ycombinator.com/item?id=22346915
This comment was briefly downvoted and flagged because the convention on HN is only to link to past threads that have comments. Otherwise users click on the link, don't find the discussion they expected, and get ornery.
Since you're new and likely didn't know this, I've restored your post. (And welcome!)
Re: Materialize: A Streaming Data Warehouse
#8Earlier quoted context omitted.
It looks like a caching bug somewhere in the web pipeline. It isn't running on Materialize, sadly. There are a pile of known limitations, and our goal is to be 100% clear about them.
Hi Frank! Congrats on the progress - I remember listening to you describing Materialized at SIGMOD'19 in the big hall - it sounded like magic back then, glad to see it come true :) I have a question now - can you subscribe to view updates somehow, instead of querying them on an ad-hoc basis?
Yes, the term we're using is "Sink", dual to a Source.
https://materialize.io/docs/sql/create-sink/
Re: Materialize: A Streaming Data Warehouse
#9Would it be fair to say this is a more OLAP-oriented approach to what KSqlDB (not KSql, but https://ksqldb.io/) does?
Seems that it's perhaps lacking the richness of how ksqldb uses Kafka Connectors (sinks and sources), but I don't see any reason you couldn't use Materialize in conjunction with ksqldb.
Eg:
KC-source --> ksql --> materialize --> kafka --> KC-sink
Question to Materialize...
What connectors (sinks and sources) do you have or plan to develop? Seems like it's mostly Kafka in and out at the moment.
Why would I use this over KSqlDB?
Can I snapshot and resume from the stream? Or do I need to rehydrate to re-establish state?
Re: Materialize: A Streaming Data Warehouse
#10Earlier quoted context omitted.
Hi Frank! Congrats on the progress - I remember listening to you describing Materialized at SIGMOD'19 in the big hall - it sounded like magic back then, glad to see it come true :) I have a question now - can you subscribe to view updates somehow, instead of querying them on an ad-hoc basis?
Yes, the term we're using is "Sink", dual to a Source. https://materialize.io/docs/sql/create-sink/ At the moment you can create Kafka sinks, and we'll write the change log out to that topic. They are still baking though, and need some hardening for failure cases.
Thanks.
What happens when a DDL is run on the source table (add or remove column for instance) ? Does it break the entire flow?