Live data from Hacker News

Postgres LISTEN/NOTIFY actually scales

dbos.dev

31–40 of 99 posts

Re: Postgres LISTEN/NOTIFY actually scales

#31
post #15
post #7

Earlier quoted context omitted.

> 5 orders of magnitude too small for another. Nitpick on an otherwise good post, but I don’t think there are very many 6billion RPS systems out there, and those that do exist are almost certainly using bespoke, purpose-built tools

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

DynamoDB scales much more than that. In one of their dynamodb papers they claimed that the amazon us retail website alone made like 89 millions rps during prime day, a few years ago.

Re: Postgres LISTEN/NOTIFY actually scales

#32

I went through this process when I was designing the sync server for Digital Carrot. In the end, I decided to just go with the simplest solution possible. In my case it's just a barebones Go gRPC service that uses an in memory channel to send notifications between connected clients. The reality is that this simple Go server will scale up to about 1000 simultaneously connected customers on about 2gb of RAM. I don't ex…

1000 connections is a fairly low number by modern standards, c10k “challenges” are like twenty years old by now.

The real questions are:

- how many messages per second are you processing on that 2gb machine (and using how many cpus)?

- does your message processing involve transaction handling, including saving data ti disk durably?

No offense but it really seems you’re comparing apples and oranges, with your use case being much much simpler than the one described.

Re: Postgres LISTEN/NOTIFY actually scales

#35
post #19

Earlier quoted context omitted.

The core optimization is to buffer notifications in-memory and send them in a batch instead of sending them as part of every transaction. So that's a general-purpose optimization for Postgres apps using LISTEN/NOTIFY.

So this is not inside a trigger but on the app connected to pg?

Yes, per the article they “scaled” Postgres to meet their requirements by altering their usage pattern to avoid hitting the bottleneck.

Re: Postgres LISTEN/NOTIFY actually scales

#36
I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to understand and we did need consistency guarantees for this workflow and this would remove the need for yet another place data got stored and transfered.

In practice, this eventually ended up being very awkward because extending the functionality (since we had "built" it) and had to work around internal pg semantics (we should have just moved off much sooner). It also did not scale well. We ended up getting a ton of disk contention on our RDS instance in non-obvious ways, and the vacuum runs on that table was a nightmare. Additionally, it was hard to get other engineers to really debug and take ownership of the system because they automatically viewed a queue (very easy to understand) implemented in a foreign way (off pg internals) as something "scary". It was emotional, not rational, but we are emotional beings, and I do not blame them. These were good engineers with a lot of other things on their plates.

Obviously, this is all hand-wavy without discussing the internal schema, indeces, etc. that we had set up, but my main takeaway with core technology from this experience was to always reach for the dumb, expected, simple thing. Even if it adds another moving piece in the infra stack. Unless I need very strong data consistency guarantees, it's always better to use something like SQS, Redis queues, etc. where the understanding is that it is just a queue (or at least the API contract suggests simplicity), and then everything needs to work around it.

The fewer mechanistic responsibilities per core data store, the better in my experience.

Re: Postgres LISTEN/NOTIFY actually scales

#38

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

It is so rare that a new moving part beats other factors. But the one thing I will say is that there's tremendous advantages to of all things, your queuing system being independent of other architectural pieces : it's the one component built to natively store and forward, so if you can keep its lifecycle separate then you can harness that to decouple other systems from each other during updates, troubleshooting, patch windows etc.

Re: Postgres LISTEN/NOTIFY actually scales

#39

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

Not saying this in a mean way:

It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to.

As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks and adding some when you actually need them makes for healthier systems.

Re: Postgres LISTEN/NOTIFY actually scales

#40
post #38

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

It is so rare that a new moving part beats other factors. But the one thing I will say is that there's tremendous advantages to of all things, your queuing system being independent of other architectural pieces : it's the one component built to natively store and forward, so if you can keep its lifecycle separate then you can harness that to decouple other systems from each other during updates, troubleshooting, patc…

Great point.

> It is so rare that a new moving part beats other factors

Fair! I will restate that we were growing very fast, so that's probably an outlier environmental factor that I am potentially overly discounting. If that's not the case (likely for most startups), then maybe this is less of a problem.

Post reply on HN