Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The notifier pattern for applications that use Postgres

brandur.org

Post reply on HN
Print view View original

11–20 of 56 posts

Re: The notifier pattern for applications that use Postgres

#11

Post by brandur »

mcqueenjordan wrote: post #10

Other than the space for past notifications and/or having to issue a DELETE, are there significant reasons to prefer this over the typical table-based approach with SKIP LOCKED queries to poll the queue? It seems to me that if the listener dies, notifications in the meantime will be dropped until a listener resubscribes, right? That seems prone to data loss. In the SKIP LOCKED topic-poller style pattern (for example,…

In Postgres listen/notify are inherently lossy channels — if a notification goes out while a listener wasn't around to receive it, it's gone, so they should never be relied upon in cases where data consistency is at stake.

I find that the main thing they're useful for is notifying on particular changes so that components that care about them can decrease the time until they process those changes, and without sitting in a hot loop constantly polling tables.

For example, I wrote a piece here [1] describe how we use the notifier to listen for feature flag changes so that each running program can update its flag cache. Those programs could be sitting in loops reloading flags once a second looking for changes, but it's wasteful and puts unnecessary load on the database. Instead, each listens for notifications indicating that some flag state changed, then reloads its flag cache. They also reload every X seconds so that some periodic synchronization happens in case an update notification was missed (e.g. a notifier temporarily dropped offline).

Job queues are another example. You'll still be using `SKIP LOCKED` to select jobs to work, but listen/notify makes it faster to find out that a new job became available.

[1] https://brandur.org/fragments/instant-feature-flags

Re: The notifier pattern for applications that use Postgres

#12

Post by thom »

mcqueenjordan wrote: post #10

Other than the space for past notifications and/or having to issue a DELETE, are there significant reasons to prefer this over the typical table-based approach with SKIP LOCKED queries to poll the queue? It seems to me that if the listener dies, notifications in the meantime will be dropped until a listener resubscribes, right? That seems prone to data loss. In the SKIP LOCKED topic-poller style pattern (for example,…

The only tradeoff here is the pure NOTIFY approach (if you don't care about losing notifications) can sit there on a single connection, and probably performs a bit better than having a bunch of workers in contention for that connection (at which point you don't really need SKIP LOCKED anyway). But ultimately tuning the level of parallelism of your worker pool and how many connections to dedicate to it doesn't seem a huge hardship.

Re: The notifier pattern for applications that use Postgres

#13

Post by santicalcagno »

mcqueenjordan wrote: post #10

Other than the space for past notifications and/or having to issue a DELETE, are there significant reasons to prefer this over the typical table-based approach with SKIP LOCKED queries to poll the queue? It seems to me that if the listener dies, notifications in the meantime will be dropped until a listener resubscribes, right? That seems prone to data loss. In the SKIP LOCKED topic-poller style pattern (for example,…

I implemented a queue using both LISTEN/NOTIFY for notifications to the task processor and SKIP LOCKED to sift through the pending tasks in the tasks table.

I think you can eliminate polling if you don't need to retry tasks, by simply processing pending tasks at startup and then just responding to LISTEN events. However, I'm curious if there are any alternatives to polling the queue in cases where you need to support retrying tasks at a given timestamp.

Re: The notifier pattern for applications that use Postgres

#14

Post by koolba »

brandur wrote: post #8
koolba wrote: post #6

This post misses the most important part of LISTEN / NOTIFY: transactions The notify piece respects your current transactional state. So of you issue a notify within a transaction, it’s only delivered if the transaction commits. If the transaction rolls back, the notification is discarded. This leads to the common pattern of combining the notification with insertion into a work queue. That solves the “listener was no…

Author here. The behavior of notify with respect to transactions is indeed notable, and definitely a great feature that makes them distinct from pub/sub in other systems. Notifies fire only when data is ready after the transaction is committed, and they're also deduplicated based on payload so listeners don't have to react to many of the same message unnecessarily. That said, NOTIFY isn't really what this post is abo…

Ha! I just realized that the “notifier” in the post title refers to the handling on the client side.

Re: The notifier pattern for applications that use Postgres

#15

Post by mcqueenjordan »

brandur wrote: post #11
mcqueenjordan wrote: post #10

Other than the space for past notifications and/or having to issue a DELETE, are there significant reasons to prefer this over the typical table-based approach with SKIP LOCKED queries to poll the queue? It seems to me that if the listener dies, notifications in the meantime will be dropped until a listener resubscribes, right? That seems prone to data loss. In the SKIP LOCKED topic-poller style pattern (for example,…

In Postgres listen/notify are inherently lossy channels — if a notification goes out while a listener wasn't around to receive it, it's gone, so they should never be relied upon in cases where data consistency is at stake. I find that the main thing they're useful for is notifying on particular changes so that components that care about them can decrease the time until they process those changes, and without sitting…

Got it, thanks for the reply. The feature flag cache reload use case seems like reasonable one to me.

Re: The notifier pattern for applications that use Postgres

#16

Post by mcqueenjordan »

santicalcagno wrote: post #13
mcqueenjordan wrote: post #10

Other than the space for past notifications and/or having to issue a DELETE, are there significant reasons to prefer this over the typical table-based approach with SKIP LOCKED queries to poll the queue? It seems to me that if the listener dies, notifications in the meantime will be dropped until a listener resubscribes, right? That seems prone to data loss. In the SKIP LOCKED topic-poller style pattern (for example,…

I implemented a queue using both LISTEN/NOTIFY for notifications to the task processor and SKIP LOCKED to sift through the pending tasks in the tasks table. I think you can eliminate polling if you don't need to retry tasks, by simply processing pending tasks at startup and then just responding to LISTEN events. However, I'm curious if there are any alternatives to polling the queue in cases where you need to support…

I personally think polling the queue/table via queries is a very sensible pattern and not something I have a desire to remove. In theory, you could go at it via a push approach by wiring into the WAL or something but that comes with its own rats nest of issues.

Re: The notifier pattern for applications that use Postgres

#18

Post by salojoo »

brandur wrote: post #9
salojoo wrote: post #7

"Instead, a received notification is immediately sent into buffered channel, which means it’s discarded if the channel is full" Shouldn't the channel rather block than discard if full? Anyway, nice and relevant article for me as I've recently added a few listeners to my app. I chose the naive approach since I only have two topics and a surplus of connections

Author here. The Go channel send behavior could certainly be altered depending on the particular semantics of the application, but the reason I chose to use a non-blocking buffered channel is so that no particular subcomponent can slow down the distribution of notifications for everybody. > Shouldn't the channel rather block than discard if full? In Go, a blocking channel is one that's initialized without a size (see…

We probably both understand the underlying concepts correctly. But just in case: also buffered channels will block if full.

So I don't see how notifications get discarded (meaning lost). But somehow that's what your text says?

Re: The notifier pattern for applications that use Postgres

#19

Post by John23832 »

brandur wrote: post #3
John23832 wrote: post #2

I thought that listen was bad because it holds connections unnecessarily and defeats connection pooling.

Listen does hold a connection, but that doesn't mean it defeats connection pooling. That's what I was trying to convey in this blog post: you'll keep a fixed number of connections open for use with listen, but as few as possible by reusing a single connection per program to simultaneously listen on all channels that your application cares about (with the notifier distributing messages to each internal component that…

Thank you. The more you know.

Re: The notifier pattern for applications that use Postgres

#20

Post by brandur »

salojoo wrote: post #18
brandur wrote: post #9

Earlier quoted context omitted.

Author here. The Go channel send behavior could certainly be altered depending on the particular semantics of the application, but the reason I chose to use a non-blocking buffered channel is so that no particular subcomponent can slow down the distribution of notifications for everybody. > Shouldn't the channel rather block than discard if full? In Go, a blocking channel is one that's initialized without a size (see…

We probably both understand the underlying concepts correctly. But just in case: also buffered channels will block if full. So I don't see how notifications get discarded (meaning lost). But somehow that's what your text says?

They'll get lost when using a non-blocking send with select/default:

    messages := make(chan string)
    select {
    case messages
The reason you'd use a non-blocking send is to make sure that in the event of one slow consumer that the entire system doesn't slow down.

Imagine a scaled out version of the notifier in which it's listening on hundreds of topics and receiving thousands of notifications. Each notification is received one-by-one using something like Pgx's `ListenForNotification`, and then distributed via channel to subscriptions that were listening for it.

In the case of a blocking send without `default`, one slow consumer that was taking too much time to receive and process its notifications would cause a build up of all other notifications the notifier's supposed to send, so one bad actor would have the effect of degrading the time-to-receive for all listening components.

With buffered channels, a poorly written consumer could still drop messages for itself, which isn't optimal (it should be fixed), but all other consumers will still receive theirs promptly. Overall preferable to the alternative.

Post reply on HN