Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY actually scales

dbos.dev

Post reply on HN
Print view View original

41–50 of 99 posts

Re: Postgres LISTEN/NOTIFY actually scales

#41

Post by vhiremath4 »

hmaxdml wrote: post #39
vhiremath4 wrote: post #36

I once was the CTO of a company that serviced about 100k requests per day across all our services. We grew to millions and eventually 10's of millions, but, somewhere along the way, an engineer on our team decided he wanted to build a queue off LISTEN/NOTIFY semantics in order to take advantage of strong consistency with the rest of our data model, which seemed reasonable given LISTEN/NOTIFY is not that hard to under…

Not saying this in a mean way: It seems to me this can be boiled down to "using things without understanding how they work doesn't scale". Yes, vanilla listen/notify doesn't scale. But the OP actually figured out how to make it scale. So your engineers don't have to. As the community builds, over time, distributed systems, it also understands which brick can do what, and it turns out that starting with less bricks an…

Nah this isn't mean at all. I think this is the correct takeaway. I actually felt like I understood how the system worked and it wasn't that difficult for me, but I also understood how, as we were adding engineers who were all very under water, the idea of learning our queueing system to modify a core feature it powered seemed like a big mental context switch (bigger than it actually was). I noted this in another comment, but I am likely discounting how big of an effect our scale/growth impacted peoples' ability to adapt this system. That said, I will pretty much always choose the simple/dumb/easy to understand thing from the start even if it adds another moving component.

Re: Postgres LISTEN/NOTIFY actually scales

#42

Post by hxtk »

acaloiar wrote: post #11

Much of the FUD around using LISTEN/NOTIFY in production comes from people who have never done so. It of course has its limits, and we should remain aware of them. Much like the limits of every piece of tech in our stacks. Choose database queue technology https://news.ycombinator.com/item?id=37636841

Personally I love it for cache invalidation. I used it on an older especially on project originally built without caching in mind that then added caching for immutable data without caching for mutable data in mind. It fell to me to add caching of mutable data. I found it rather convenient to put NOTIFY triggers on cached tables and have a LISTENer that will delete the corresponding rows from memcached when it gets a notification. It’s basically impossible to out-scale it in that use case because the scale is intrinsically linked to number of writes the leader in your database processes.

Re: Postgres LISTEN/NOTIFY actually scales

#43

Post by newswangerd »

znpy wrote: post #32
newswangerd wrote: post #28

I went through this process when I was designing the sync server for Digital Carrot. In the end, I decided to just go with the simplest solution possible. In my case it's just a barebones Go gRPC service that uses an in memory channel to send notifications between connected clients. The reality is that this simple Go server will scale up to about 1000 simultaneously connected customers on about 2gb of RAM. I don't ex…

1000 connections is a fairly low number by modern standards, c10k “challenges” are like twenty years old by now. The real questions are: - how many messages per second are you processing on that 2gb machine (and using how many cpus)? - does your message processing involve transaction handling, including saving data ti disk durably? No offense but it really seems you’re comparing apples and oranges, with your use case…

These are all very conservative back of the napkin calculations. Also, this isn't 1000 connections. It's 1000 customers, each of which can consume dozens of connections.

My point here is that it is important to match the tech stack to the challenge you're facing. When I started thinking about how to solve this problem my first reaction was to design an overly complicated distributed message queue using PG Notify, Redis, Kafka or something along those lines. The key takeaway here is that I realized that I probably wouldn't end up with more than 1000 customers, so I just needed to design a system that could comfortably handle that level of traffic without much effort. If, by some miracle, my business goes crazy viral, I know that my cloud provider can probably handle up to 200,000 customers by just updating a slider in my dashboard, which is way more business than I want anyway.

Engineers love to fantasize about Google levels of scale, but that's just not realistic for a lot of services.

Re: Postgres LISTEN/NOTIFY actually scales

#44

Post by [deleted] »

znpy wrote: post #31
odo1242 wrote: post #15

Earlier quoted context omitted.

The highest I could think of is WhatsApp, which gets ~1.6 million RPS on average, and they use MQTT

DynamoDB scales much more than that. In one of their dynamodb papers they claimed that the amazon us retail website alone made like 89 millions rps during prime day, a few years ago.

[deleted]

Re: Postgres LISTEN/NOTIFY actually scales

#45

Post by chmod775 »

ndriscoll wrote: post #25
monster_truck wrote: post #23

Earlier quoted context omitted.

Maybe there would be more if it were more straightforward to do so?

For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS. There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet.

> There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet.

Roughly a hundred servers on 100 GbE ports each could handle one small UDP packet per human every second, and would have a plenty of cycles and RAM per human left to actually do some work.

That fits in a shipping container.

Illustrative of course: the point is that "for every human" is not far out there given how crazy powerful hardware has become.

Re: Postgres LISTEN/NOTIFY actually scales

#46

Post by elendilm »

sandeepkd wrote: post #20

I think a lot of these kind of posts are standalone assessment of your problems, understanding and solutions. Its debatable to term something as lack of expertise if one is trying to work with default settings of a tool and expecting a certain performance. Everyone is doing a continuous learning with the failures. 1. What I find interesting is that the experiment seems to be using a DB server with 96 cores, 384 GB RA…

Agreed

Re: Postgres LISTEN/NOTIFY actually scales

#47

Post by elendilm »

The tested machine appears big and with 96 vcpu + 384 gb yielding 20k writes sounds too low. Jumping to 60k is very good. But still too small a throughput for what the machine is capable of.

Batching ensures you run at cpu & memory speeds and only pay significant latency for the flush - which usually linux kernel coalesces well if concurrent.

Re: Postgres LISTEN/NOTIFY actually scales

#48

Post by AdieuToLogic »

chmod775 wrote: post #45
ndriscoll wrote: post #25

Earlier quoted context omitted.

For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS. There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet.

> There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet. Roughly a hundred servers on 100 GbE ports each could handle one small UDP packet per human every second, and would have a plenty of cycles and RAM per human left to actually do some work. That fits in a shipping container. Illustrative of course: the point is that "for every human" is not far out the…

>> For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS.

> Roughly a hundred servers on 100 GbE ports each could handle one small UDP packet per human every second ...

While there is no correlation between the Visa claim of 70k tps to people on the planet, there is also no correlation to the number of servers needed to process UDP packets.

The "Visa network" is more of a trusted collaboration between a small number of banks operating multi-step transactions than it is an OSI Layer 1 - 5 concern.

Re: Postgres LISTEN/NOTIFY actually scales

#49

Post by chmod775 »

AdieuToLogic wrote: post #48
chmod775 wrote: post #45

Earlier quoted context omitted.

> There are not very many systems one could conceive of that could do useful work 1/s for ~every human on the planet. Roughly a hundred servers on 100 GbE ports each could handle one small UDP packet per human every second, and would have a plenty of cycles and RAM per human left to actually do some work. That fits in a shipping container. Illustrative of course: the point is that "for every human" is not far out the…

>> For reference, I've seen Visa marketing materials that suggest their network can do ~70k TPS. > Roughly a hundred servers on 100 GbE ports each could handle one small UDP packet per human every second ... While there is no correlation between the Visa claim of 70k tps to people on the planet, there is also no correlation to the number of servers needed to process UDP packets. The "Visa network" is more of a truste…

> Illustrative of course: the point is that "for every human" is not far out there given how crazy powerful hardware has become.
Post reply on HN