Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY actually scales

dbos.dev

Post reply on HN
Print view View original

81–90 of 99 posts

Re: Postgres LISTEN/NOTIFY actually scales

#81

Post by serhii777_123 »

BiteCode_dev wrote: post #67
vhiremath4 wrote: post #41

Earlier quoted context omitted.

Nah this isn't mean at all. I think this is the correct takeaway. I actually felt like I understood how the system worked and it wasn't that difficult for me, but I also understood how, as we were adding engineers who were all very under water, the idea of learning our queueing system to modify a core feature it powered seemed like a big mental context switch (bigger than it actually was). I noted this in another com…

Also the results given it the article are still worse than redis with zero work and no risk of breaking later, both in nb of req and latency. And you can move redis to another server next later if needed to split the db and notification load.

[flagged]

Re: Postgres LISTEN/NOTIFY actually scales

#82

Post by dr_dshiv »

illliillll wrote: post #60
odo1242 wrote: post #15

Earlier quoted context omitted.

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

I used to scrape Meta sites at over 200k steady rps for a couple of years, all by myself. They did send lawyers to my door in the end though. Someone had forgot IPv6 rate limit bucketing :)

Tell more sounds like a good story

Re: Postgres LISTEN/NOTIFY actually scales

#83

Post by rlio »

the article mentions lock contention on the global queue, but i think doesn’t mention the other problem with a fixed size global queue: a single slow reader on one channel can block all writes to all channels. (at least this was a failure mode a few years ago! perhaps it’s changed since i last looked)

Re: Postgres LISTEN/NOTIFY actually scales

#84

Post by Latty »

pgwhalen wrote: post #77
Latty wrote: post #3

One way it explicitly doesn't scale (unless something has changed since I last checked and my quick search failed me) is the hard limit on 8000 bytes of data in a notification. If your notification doesn't make sense to exist as a row (where you can just give an ID), then that makes it hard to use. I had a web game and the events were transient descriptions of changes in state that didn't make sense to store in the d…

Could you not send a message that pointed to the state that changed in a non-row-specific way?

It was diffs in state that got sent to the client, so recomputing them from the current state would have required having the previous state.

Re: Postgres LISTEN/NOTIFY actually scales

#85

Post by rneswold »

hoodaly wrote: post #63
phamilton wrote: post #51

Just sharing a data point and experience. We had a lot of success with LISTEN/NOTIFY when we paired it with a Rust graphql subscription broker. 10s of thousands of subscriptions, but only 3 or 4 LISTEN connections (one for each host). All changes would be pushed out to all hosts, who would each manage the actual user subscriptions and choose what to actually publish. This worked super well. In general, moving from hu…

I‘m interested, which graphql library for Rust can you recommend?

I like 'async-graphql'. I tried 'juniper' but, at the time, it didn't have as many features. That may have changed over the last year or so since I looked at it.

Re: Postgres LISTEN/NOTIFY actually scales

#86

Post by phamilton »

hoodaly wrote: post #63
phamilton wrote: post #51

Just sharing a data point and experience. We had a lot of success with LISTEN/NOTIFY when we paired it with a Rust graphql subscription broker. 10s of thousands of subscriptions, but only 3 or 4 LISTEN connections (one for each host). All changes would be pushed out to all hosts, who would each manage the actual user subscriptions and choose what to actually publish. This worked super well. In general, moving from hu…

I‘m interested, which graphql library for Rust can you recommend?

Any graphql library would work. I like async-graphql except compilation is a bear due to all the macros. We wired LISTEN up to subscriptions using tokio mpmc channels. It was a great moment of "fearless concurrency", it all just worked.

Re: Postgres LISTEN/NOTIFY actually scales

#87

Post by ndriscoll »

stavros wrote: post #75
ndriscoll wrote: post #25

Earlier quoted context omitted.

For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS. There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet.

Hm, what am I not getting? One transaction per second for every human on the planet would be 8 billion TPS, not 70k, no?

Yes, the point is it's difficult to even conceive of something that needs anywhere near the billions of TPS, because generally speaking, humans are not doing something with a computer once per second on average. Far lower. I might use my credit card less than once per day on average.

Re: Postgres LISTEN/NOTIFY actually scales

#88

Post by paulddraper »

monster_truck wrote: post #23
zbentley wrote: post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

Maybe there would be more if it were more straightforward to do so?

Doing 1 request for every person on this planet every second?

What are these, TCP packets?

Re: Postgres LISTEN/NOTIFY actually scales

#89

Post by stavros »

ndriscoll wrote: post #87
stavros wrote: post #75

Earlier quoted context omitted.

Hm, what am I not getting? One transaction per second for every human on the planet would be 8 billion TPS, not 70k, no?

Yes, the point is it's difficult to even conceive of something that needs anywhere near the billions of TPS, because generally speaking, humans are not doing something with a computer once per second on average. Far lower. I might use my credit card less than once per day on average.

Ah, yes, humans don't, but computers do. I'd imagine there are plenty of systems that ingest automated data (eg sensors) that need billions of TPS.

Re: Postgres LISTEN/NOTIFY actually scales

#90

Post by winrid »

vhiremath4 wrote: post #36

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

Pretty sure if you didn't run the DB on networked disks you would have been fine.
Post reply on HN