Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY actually scales

dbos.dev

Post reply on HN
Print view View original

51–60 of 99 posts

Re: Postgres LISTEN/NOTIFY actually scales

#51

Post by phamilton »

Just sharing a data point and experience.

We had a lot of success with LISTEN/NOTIFY when we paired it with a Rust graphql subscription broker. 10s of thousands of subscriptions, but only 3 or 4 LISTEN connections (one for each host). All changes would be pushed out to all hosts, who would each manage the actual user subscriptions and choose what to actually publish.

This worked super well. In general, moving from hundreds of Ruby or Node hosts to just a few Rust hosts just allows so many simplifications and things that "don't scale" to actually work quite well.

Re: Postgres LISTEN/NOTIFY actually scales

#52

Post by ksec »

ndriscoll wrote: post #25
monster_truck wrote: post #23

Earlier quoted context omitted.

Maybe there would be more if it were more straightforward to do so?

For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS. There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet.

Considering Visa's user Time zone and population density. I wouldn't be surprised if WeChat in China or UPI in India have higher TPS numbers.

But then again it would still be no where near a million TPS.

Re: Postgres LISTEN/NOTIFY actually scales

#53

Post by JoelJacobson »

KraftyOne wrote: post #12
dietr1ch wrote: post #2

I recall that in the first release that supported LISTEN/NOTIFY there was a performance issue around it (poor locking IIRC), which today the "bad post" mentioned in here corrects in a errata just after their first paragraph. Since the correction apparently dates from May 8th, I think that a post from July 24th might want to acknowledge that the popular post asserting this feature doesn't (didn't?) scale was not made…

If you mean the optimizations coming in Postgres 19, the original post addresses this: > As an aside, there’s been some online discussion of a Postgres patch ( https://github.com/postgres/postgres/commit/282b1cde9dedf456... ) related to this issue. This patch (to be released in Postgres 19) does not remove the global lock or fix the bottleneck we observed. Instead, it optimizes the narrower case where there are many…

It did fix the original "Postgres LISTEN/NOTIFY does not scale" [1] post's problem though, which was mentioned in an update of that article:

    Update: Fixed in Postgres core
    This commit has eliminated the bottleneck in the postgres core.
    Credit to Joel Jacobson and the core postgres contributors for resolving this.
[1] https://www.recall.ai/blog/postgres-listen-notify-does-not-s...

Re: Postgres LISTEN/NOTIFY actually scales

#54

Post by stingraycharles »

odo1242 wrote: post #15
zbentley wrote: post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

I work in manufacturing, you would be surprised at the volumes of sensor data being processed there for predictive maintenance.

MQTT is everywhere, indeed, because of how simple the protocol is (it’s very ubiquitous in IoT / embedded systems).

Re: Postgres LISTEN/NOTIFY actually scales

#58

Post by dagss »

I feel like the article is leaving the most important part out ; how to allocate that "offset" (sequence number) which allows consumers to keep tabs on how far they have read and query if there are new messages in the fallback. There are a few schemes, some more creative than others, but it is not a trivial problem to solve without complexity or possibility of lock contention (or races with consumers if you do it the wrong way). Most people I guess have writers acquire a lock, for instance on a single row in a state table, for allocating the next sequence number for an event topic.

What is the best way? Perhaps a tool that reads CDC and writes allocated event sequence numbers to another table would work ok, or would that have long latency?

And that CDC processor should then do the NOTIFY too.

Also on the consumer side in some usecases batching can drastically improve throughput. You don't even use LISTEN/NOTIFY then. Just run the consumer in a loop and each iteration process all new unprocessed messages, and store your last sequence number processed between iterations.

Re: Postgres LISTEN/NOTIFY actually scales

#59

Post by [deleted] »

jerf wrote: post #4

"Scale" isn't a binary, it's a continuum. "Scales to 60K/s" can be 5 orders of magnitude more than one system needs and 5 orders of magnitude too small for another. Personally I'd knock the general "premature optimization" off the list of "most common developer errors" and put in its place "using techs with the wrong scaling factors". If you use something too small and you exceed its needs, the failure is obvious, bu…

[deleted]

Re: Postgres LISTEN/NOTIFY actually scales

#60

Post by illliillll »

odo1242 wrote: post #15
zbentley wrote: post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

I used to scrape Meta sites at over 200k steady rps for a couple of years, all by myself.

They did send lawyers to my door in the end though.

Someone had forgot IPv6 rate limit bucketing :)

Post reply on HN