51–60 of 108 posts
If you have a query you could reorientate the table's changes and see if the changes affect the output incrementally manually.
PipelineDB might be of interest. https://github.com/pipelinedb/pipelinedb
Disclaimer: I work at Materialize. This is the exact problem that we are solving here at Materialize! I wrote an example blogpost that details to how to subscribe to a SQL query in practice: https://materialize.com/a-simple-and-efficient-real-time-app... Regarding your comment about "focus on streams", it's true that we first focused on extracting event data from other sources (Kafka, Debezium, direct change data cap…
I am considering Materialize for a project, and I'm concerned about the limitation that materialized views should fit into the memory of a single host. My use case is a materialized OLAP "fact" tables, created by joining a 5-10 separate tables. While the biggest source tables have under a million rows, the joined fact table would have perhaps 100 million rows which I fear might be too big. One workaround I considered…
Do you have a limit in mind on how large you're willing to go? A brief bit of napkin math shows that, at 1KB per record, your largest view should fit within 100GB of memory. This is easily supported by Materialize and I can certainly understand if this exceeds your appetite!
If you're interested in reducing the size of the in-memory dataset, does your fact table have a temporal dimension to it? By default, Materialize stores data for all time (like a database) and you can write views in such a way that it will only materialize recent data (like a stream processor). Our co-founder Frank wrote a blog post[1] detailing how to do this.
I'm honestly not sure how TAIL-ing a non-materialized view would perform. Sounds like something that would be fun to test!
Happy to chat about this further in our community Slack channel if you have more questions.
Earlier quoted context omitted.
I believe hasura uses polling under the hood to accomplish subscriptions, right?
I work at Hasura. Hasura uses a novel way of batching similar parameterized subscriptions together and then polls under the hood. This means that if there are 1000 subscribers of similar type of query, then underneath Hasura will only make a single query to Postgres (or few queries depending on the batch size). This approach, which we call "multiplexing" in short, scales really really well. And is also the simplest w…
I'm not sure I would call this relatively easy even without joins. Without joins Google Cloud Firestore does exactly what you're describing. The initial query runs against the DB then the client gets a filtered stream of updates to only that query. Its distributed and scales logarithmically with the number of queries, as it doesn't need to keep the query results in memory/materialized. The fun part of this problem is…
You can listen per code and notify per code.
https://www.postgresql.org/docs/current/sql-listen.html
The js adapter for pg can handle this: https://medium.com/@simon.white/postgres-publish-subscribe-w...
The first thing that comes to mind ist LISTEN/NOTIFY from postgres. You can listen per code and notify per code. https://www.postgresql.org/docs/current/sql-listen.html The js adapter for pg can handle this: https://medium.com/@simon.white/postgres-publish-subscribe-w...
We have an in house system (LunaDb) which is a little like this. There's a tech talk available about how it works at https://blog.asana.com/2015/10/asana-tech-talk-reactive-quer... - it's from a few years ago, but the core ideas are there. There's also some details on the caching layer we built for it at https://blog.asana.com/2020/09/worldstore-distributed-cachin...
A few properties based on your questions and the observations here:
- We don't attempt to incrementally update query results. Given the number of simultaneous queries the system handles, we've found it much more important to instead by very precise about only re-running exactly the right queries in response to data modification.
- We support joins (although not queries of arbitrary complexity). We avoid a race conditions and cross-table locking issues by using the binlog as our source of changes, which imposes a linearization on the changes matching the state in the db. Correctly determining which queries to update for these requires going back to the database.
- Performance is an interesting problem. It's easy to arrange situations where total load is a function of "rate of data change" * "number of queries over that data", so being overly broad in triggering recalculations gets expensive fast.
We're actively hiring to work on this - if you are interested my contact details are in my hn profile.
I'm not sure I would call this relatively easy even without joins. Without joins Google Cloud Firestore does exactly what you're describing. The initial query runs against the DB then the client gets a filtered stream of updates to only that query. Its distributed and scales logarithmically with the number of queries, as it doesn't need to keep the query results in memory/materialized. The fun part of this problem is…
- the query language is quite limited (fe, searching on the absence of an attribute is not possible)
- shame you can't do projections on the changes server side.
you get the whole thing and not just the fields you're interested in