Live data from Hacker News

Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

news.ycombinator.com

31–40 of 108 posts

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#31
post #10
post #5

> I guess it's relatively easy to do This is a very hard problem to do the right way and probably would need some changes on the RDBMS itself. You would need to monitor all tables that might affect your query for changes and how these changes affect your query (say you're just reading a value, aggregating with sum, doing average with count of rows, the list goes on). Add more complexity on top of that if you want to…

> the right way Yep. I meant it was easy to do it the inefficient way where you just refresh the entire query when any table mentioned in the query changes. You would just have to also check if something was a view and recursively parse the SQL that is used in the view. Just use Postgres `LISTEN` and triggers or the WAL for change monitoring. > how these changes affect your query Yeh, this is where it gets tricky. I…

> Yep. I meant it was easy to do it the inefficient way where you just refresh the entire query when any table mentioned in the query changes.

You might find it’s cheaper to do just that. It might cost you in computational resources, but save you millions in engineering time.

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#32
Can you speak more to how updates get pushed out to your UI clients? Are you using Firestore ? Hasura? Polling?

Something we're looking at enabling w/ Estuary Flow [1] is materializing to Firestore for this use case (and Flow already supports PostgreSQL materializations).

Flow would capture changes from your database tables (or you write normalized events directly to Flow collections). You aggregate and join as needed, and then materialize your views into Firestore (or another Postgres table) where you'd subscribe for live updates.

[1]: https://github.com/estuary/flow

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#33
Informix has its "push data" feature - except it appears to be limited to a single table, not complex queries - I suspect you're right, it's hard to do safely and efficiently.

https://www.ibm.com/docs/en/informix-servers/14.10/14.10?top...

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#34
post #6

Disclaimer: I work at Materialize. This is the exact problem that we are solving here at Materialize! I wrote an example blogpost that details to how to subscribe to a SQL query in practice: https://materialize.com/a-simple-and-efficient-real-time-app... Regarding your comment about "focus on streams", it's true that we first focused on extracting event data from other sources (Kafka, Debezium, direct change data cap…

Materialize definitely looks interesting. I'd have a few questions regarding the current product:

Does Materialize provide a commercial non-cloud version where workers can run on multiple nodes in a cluster? On the website I only see references to the BSL-licensed single-node version and to the cloud version that is hosted at Materialize. Would a company be able to run Materialize on their own Kubernetes cluster?

Also is it possible to add custom code for data sources (for example Kafka, but with a different format than what Materialize currently expects) or is Materialize limited to the pre-defined sources?

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#36
I'm not sure I would call this relatively easy even without joins. Without joins Google Cloud Firestore does exactly what you're describing. The initial query runs against the DB then the client gets a filtered stream of updates to only that query. Its distributed and scales logarithmically with the number of queries, as it doesn't need to keep the query results in memory/materialized.

The fun part of this problem is that it's really inverted from most traditional database literature out there. Mostly the problem is you have a query and need to find the matching rows. To make updates efficient you need to do this the other way around - if there is a row (that has changed) find all the matching queries.

With joins (or any data dependant query where you can't tell a row is in the result set without looking at other data) you need to keep the query results materialized otherwise you can't have enough information without going back to disk or keeping everything in memory, which isn't really feasible in most cases.

Source: I worked on Google Cloud Firestore from it's launch until 2020 and was the on responsible for the current implementation and data structures of how changes get broadcasted to subscribed queries.

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#38

It's not clear to me if this would solve your problem, because it kind of depends on how you are planning on reacting, but Github and Gitlab have the concept of a Code Owner and it can work for specific files. So if you know that your queries that you want to monitor exist in tidy DB Impl classes or maybe a specific repository, you could be the code owner of those files. This approach is pro-active because you can bl…

It sounds like you're talking about a totally different question. OP is interested in subscribing to changes in the results of a query at runtime, not changes to the text of the query itself.

Sadly no. He's actually talking about using git instead of a database for storing data, which has come up a few times in the past month or so by clueless people.

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#39
A few years ago, I made node.js packages to do this for MySQL (by reading changes from the replication log) and Postgres. (using triggers) Both require specifying cache invalidation functions for each dependent table.

https://github.com/numtel/mysql-live-select

https://github.com/numtel/pg-live-select

Re: Ask HN: Is there a way to efficiently subscribe to an SQL query for changes?

#40
A sqlite database observation system in swift: https://github.com/groue/GRDB.swift#valueobservation

I don't believe it includes any efficiency magic to know "which rows" are affected. Even knowing what tables are affected required some... creative coding. A little insight into how it's implemented: https://www.mail-archive.com/sqlite-users@mailinglists.sqlit...

Post reply on HN