Any post from brandur is a must read. Love the post
The notifier pattern for applications that use Postgres
21–30 of 56 posts
Re: The notifier pattern for applications that use Postgres
#22Earlier quoted context omitted.
We probably both understand the underlying concepts correctly. But just in case: also buffered channels will block if full. So I don't see how notifications get discarded (meaning lost). But somehow that's what your text says?
They'll get lost when using a non-blocking send with select/default: messages := make(chan string) select { case messages The reason you'd use a non-blocking send is to make sure that in the event of one slow consumer that the entire system doesn't slow down. Imagine a scaled out version of the notifier in which it's listening on hundreds of topics and receiving thousands of notifications. Each notification is receiv…
Right, I didn't know it was possible to bypass full channels like that with select/default. Thanks for spelling it out
Re: The notifier pattern for applications that use Postgres
#23Any post from brandur is a must read. Love the post
Thanks Shayon!!! Very kind of you :)
Re: The notifier pattern for applications that use Postgres
#24I built a service using LISTEN/NOTIFY a while ago, but the fact that Postgres will instantly drop all messages to a given channel if there are no registered listeners made me nervous.
This felt like a big caveat to me, taking what could be an extremely well-guaranteed approach (e.g. a db trigger that writes a row to an event table, which is being polled by an application process) into an ops risk, where transient network issues could result in data loss. You’d want idempotent tasks and zero-downtime deployments, but it feels like the benefits don’t really justify the risk. If you want a topic exchange just have the polling process forward to a real message broker.
I like the concept of sending messages straight from the db, and we did run my service in production (for non-critical stuff), but unless something has changed with the way this works I would still be wary.
Re: The notifier pattern for applications that use Postgres
#25Take a look at Materialize, Noria and the family of Differential/Timely Dataflow technologies. It's the same concept on steroids, you can subscribe to arbitrary queries and efficiently receive any changes to that view. You can also efficiently maintain any materialized view for extremely fast reads for known queries.
An automatic stream processing pipeline for maintaining caches and listening to complex real-time events.
Quite underrated, it has so much promise. The concept is not new but it's still semi-stuck in Rust-land. It's becoming more mainstream with Materialize, which is technically open-source, but they are quite aggressive with pushing their expensive cloud and offuscating on-prem usage.
https://github.com/MaterializeInc/materialize
https://github.com/mit-pdos/noria
Re: The notifier pattern for applications that use Postgres
#26Take a look at Materialize, Noria and the family of Differential/Timely Dataflow technologies. It's the same concept on steroids, you can subscribe to arbitrary queries and efficiently receive any changes to that view. You can also efficiently maintain any materialized view for extremely fast reads for known queries. An automatic stream processing pipeline for maintaining caches and listening to complex real-time eve…
All cool stuff.
But to my mind, the main advantage for this postgres workflow is that you get something simple without adding a ton of stuff to your stack.
Re: The notifier pattern for applications that use Postgres
#27I built a service using LISTEN/NOTIFY a while ago, but the fact that Postgres will instantly drop all messages to a given channel if there are no registered listeners made me nervous. This felt like a big caveat to me, taking what could be an extremely well-guaranteed approach (e.g. a db trigger that writes a row to an event table, which is being polled by an application process) into an ops risk, where transient net…
When I've used it, I've used it in conjunction with a table that maintains state.
So basically the notify keeps other processes from having the poll the table for updates, but the table itself still acts as the queue.
Re: The notifier pattern for applications that use Postgres
#28Take a look at Materialize, Noria and the family of Differential/Timely Dataflow technologies. It's the same concept on steroids, you can subscribe to arbitrary queries and efficiently receive any changes to that view. You can also efficiently maintain any materialized view for extremely fast reads for known queries. An automatic stream processing pipeline for maintaining caches and listening to complex real-time eve…
All cool stuff. But to my mind, the main advantage for this postgres workflow is that you get something simple without adding a ton of stuff to your stack.
Indeed! It would be so much better if this were a Postgres extension instead.
There are some efforts but still quite immature: https://github.com/sraoss/pg_ivm
But at least Materialize does have Postgres wire compatibility, and same for Noria and MySQL. It's a plug & play switch, it's not as complex as adopting other Pub-Sub, Job-Queue or KV-Caching solutions.
Re: The notifier pattern for applications that use Postgres
#29Take a look at Materialize, Noria and the family of Differential/Timely Dataflow technologies. It's the same concept on steroids, you can subscribe to arbitrary queries and efficiently receive any changes to that view. You can also efficiently maintain any materialized view for extremely fast reads for known queries. An automatic stream processing pipeline for maintaining caches and listening to complex real-time eve…
For folks using Typescript, we're building this for web development at Triplit. We support incrementally updating subscribed queries in real-time from server to client over web-sockets.
Re: The notifier pattern for applications that use Postgres
#30Take a look at Materialize, Noria and the family of Differential/Timely Dataflow technologies. It's the same concept on steroids, you can subscribe to arbitrary queries and efficiently receive any changes to that view. You can also efficiently maintain any materialized view for extremely fast reads for known queries. An automatic stream processing pipeline for maintaining caches and listening to complex real-time eve…
(Materialize CTO here.)
> It's becoming more mainstream with Materialize, which is technically open-source, but they are quite aggressive with pushing their expensive cloud and offuscating on-prem usage.
Quick but important clarification: Materialize is source available, not open source. We've been licensed under the BSL [0] from the beginning. We feel that the BSL is the best way to ensure we can build a sustainable business to fund Materialize's development, while still contributing our research advances back to the scientific community.
> Quite underrated, it has so much promise.
I'm glad you think so. We think so too. One of the best parts of my job is watching the "aha" moment our prospects have when they realize how much of the complex code they've been writing is neatly expressed as a SUBSCRIBE over a SQL materialized view.
[0]: https://github.com/MaterializeInc/materialize/blob/main/LICE...