Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY actually scales

dbos.dev

Post reply on HN
Print view View original

91–99 of 99 posts

Re: Postgres LISTEN/NOTIFY actually scales

#91

Post by vhiremath4 »

eudamoniac wrote: post #80
vhiremath4 wrote: post #36

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

I feel this is a management problem, and I disagree with the takeaway about adding a piece to the infra stack. "My devs don't want to work on part of our stack" is not a valid reason to add a new network node imo. Just make them work on it.

What is cheaper? Forcing my team to work on something they don't want to or adding a network node (that happens to also be the industry standard)? I used to think in absolutes about these kinds of things, but I came to realize along the way that people management is a process of learning which fires to take on. If I can avoid stripping away someone's autonomy, I will always do that even if I know they are not doing the correct thing (unless the wrong thing is in the critical path of the company).

The problem still needs to be solved (queue working/being up/being built around), but I am ok with the solution that might not be the best engineering solution but still delivers on the product requirements and extends the team's happiness (these systems need to be owned). These were not lazy or incompetent engineers.

Re: Postgres LISTEN/NOTIFY actually scales

#92

Post by coreylane »

odo1242 wrote: post #15
zbentley wrote: post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

AWS IAM handles over 1.2 billion requests per second, based on a 2024 post, im sure its higher now https://aws.amazon.com/blogs/security/an-unexpected-discover...

Re: Postgres LISTEN/NOTIFY actually scales

#93

Post by gunnarmorling »

elendilm wrote: post #47

The tested machine appears big and with 96 vcpu + 384 gb yielding 20k writes sounds too low. Jumping to 60k is very good. But still too small a throughput for what the machine is capable of. Batching ensures you run at cpu & memory speeds and only pay significant latency for the flush - which usually linux kernel coalesces well if concurrent.

Yeah, with that machine spec, I think if anything, the post goes to show that LISTEN/NOTIFY does actually _not_ scale well. A CDC-based solution would give you a multiple of these numbers on much smaller (and cheaper) hardware.

Re: Postgres LISTEN/NOTIFY actually scales

#94

Post by nly »

odo1242 wrote: post #15
zbentley wrote: post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

WhatsApp is primarily 1 to 1 messaging. Why would this be hard to scale? Scaling horizontally, sharding, and message routing is all it takes, surely?

Re: Postgres LISTEN/NOTIFY actually scales

#95

Post by znpy »

newswangerd wrote: post #43
znpy wrote: post #32

Earlier quoted context omitted.

1000 connections is a fairly low number by modern standards, c10k “challenges” are like twenty years old by now. The real questions are: - how many messages per second are you processing on that 2gb machine (and using how many cpus)? - does your message processing involve transaction handling, including saving data ti disk durably? No offense but it really seems you’re comparing apples and oranges, with your use case…

These are all very conservative back of the napkin calculations. Also, this isn't 1000 connections. It's 1000 customers, each of which can consume dozens of connections. My point here is that it is important to match the tech stack to the challenge you're facing. When I started thinking about how to solve this problem my first reaction was to design an overly complicated distributed message queue using PG Notify, Red…

Engineers also love to reinvent the wheel.

What you wrote is essentially technical debt.

Re: Postgres LISTEN/NOTIFY actually scales

#96

Post by znpy »

junon wrote: post #73
znpy wrote: post #31

Earlier quoted context omitted.

DynamoDB scales much more than that. In one of their dynamodb papers they claimed that the amazon us retail website alone made like 89 millions rps during prime day, a few years ago.

It's also one of the most aggravating database systems I've worked with.

It still scales though

Re: Postgres LISTEN/NOTIFY actually scales

#97

Post by dynotable »

znpy wrote: post #31
odo1242 wrote: post #15

Earlier quoted context omitted.

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

DynamoDB scales much more than that. In one of their dynamodb papers they claimed that the amazon us retail website alone made like 89 millions rps during prime day, a few years ago.

89M rps was the peak across all Amazon systems during 2021 Prime Day. Alexa, Amazon.com and the fulfilment centres together, not one site. But its still very impressieve. But the more importarnt number is the per partition limit, 3000 RCU and 1000 WCU. Horizontal scale is the easy part, a table only gets near it if the partition key actually spreads.

Re: Postgres LISTEN/NOTIFY actually scales

#98

Post by odo1242 »

nly wrote: post #94
odo1242 wrote: post #15

Earlier quoted context omitted.

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

WhatsApp is primarily 1 to 1 messaging. Why would this be hard to scale? Scaling horizontally, sharding, and message routing is all it takes, surely?

Wasn't saying it would be hard to scale at all, I just mentioned the highest RPS customer-facing system I could think of off the top of my head.

Re: Postgres LISTEN/NOTIFY actually scales

#99

Post by odo1242 »

coreylane wrote: post #92
odo1242 wrote: post #15

Earlier quoted context omitted.

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

AWS IAM handles over 1.2 billion requests per second, based on a 2024 post, im sure its higher now https://aws.amazon.com/blogs/security/an-unexpected-discover...

Ah, interesting.
Post reply on HN