Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY does not scale

recall.ai

Post reply on HN
Print view View original

321–328 of 328 posts

Re: Postgres LISTEN/NOTIFY does not scale

#321

Post by incoming1211 »

0xbadcafebee wrote: post #108
incoming1211 wrote: post #37

Earlier quoted context omitted.

Because transactional databases are perfectly fine for this type of thing when you have 0 to 100k users.

The total number of users in your system is not a performance characteristic. And transactions are generally wrong for write-heavy anything. Further, if you can just append then the transaction is meaningless.

Most systems are based on the number of users performing operations on the application. Majority of people on HN never work on anything with more than 100k users, yet they introduce mountains of infrastructure and blame cloud for being expensive when they never needed that infrastructure to begin with.

Re: Postgres LISTEN/NOTIFY does not scale

#322

Post by dumbfounder »

incoming1211 wrote: post #37
dumbfounder wrote: post #35

Transactional databases are not really the best tool for writing tons of (presumably) immutable records. Why are you using it for this? Why not Elastic?

Because transactional databases are perfectly fine for this type of thing when you have 0 to 100k users.

The article is about scaling issues with tons of writes, which I referenced by saying “tons of writes”. Yes, if there is no reason to scale it up you can just use a database. But that’s not the context.

Re: Postgres LISTEN/NOTIFY does not scale

#323

Post by panzi »

Lio wrote: post #313
panzi wrote: post #279

Earlier quoted context omitted.

Back when I used Rails the sentiment was: You don't need foreign keys, this is all handled by ActiveRecord.

I’m guessing that must have been the 00s then. I haven’t seen anyone downplay database constraints for a very long time.

Sometime between 2005 and 2012 I think. It definitely was for a stretch of time where Rails people said that.

Re: Postgres LISTEN/NOTIFY does not scale

#324

Post by mike_hearn »

ownagefool wrote: post #319
mike_hearn wrote: post #247

Earlier quoted context omitted.

Yes that's true but in good implementations you will want to surface to the recipient via some dashboard if delivery consistently fails. So at some point a message on the exception queue will want to update the db.

Whilst true, it probably doesn't need ACID / strong consistency. Don't get me wrong. Correctness is a great default, much easier to reason about.

You're right, I think we're in agreement.

Re: Postgres LISTEN/NOTIFY does not scale

#325

Post by mike_hearn »

imtringued wrote: post #251
dathinab wrote: post #235

Earlier quoted context omitted.

if you need transaction across a queue into a normal SQL DB or similar I believe you are doing something very wrong. Sure you need transaction about processing things in a queue (mark as "taken out", but not yet remove then remove or "place back in (or into a failed messages inbox)" on timeout or similar can be _very_ important for queue systems. But the moment the "fail save if something dies while processing a mess…

>But the moment the "fail save if something dies while processing a message" becomes a directly coupled with DB transactions you have created something very brittle and cumbersome. The standard workflow for processing something from a queue is to keep track of all the messages you have already processed in the transactional database and simply request the remaining unprocessed messages. Often this is as simple as sto…

That approach doesn't parallelize, it's the kind of thing I mean when I say that custom idempotency logic is tricky. But yes, you can set things up so the queue is outside the database. It's just more convenient not to.

Re: Postgres LISTEN/NOTIFY does not scale

#327

Post by brightball »

gunnarmorling wrote: post #306
brightball wrote: post #244

Earlier quoted context omitted.

You know, this would be a great talk at the 2026 Carolina Code Conference...

Ha, that's interesting :) Do you have any more details to that one?

Traveling now, but I’ll connect when I get back

Re: Postgres LISTEN/NOTIFY does not scale

#328

Post by brightball »

brightball wrote: post #327
gunnarmorling wrote: post #306

Earlier quoted context omitted.

Ha, that's interesting :) Do you have any more details to that one?

Traveling now, but I’ll connect when I get back

I don't see any contact info in your profile, but we typically open the call for speakers from early January - end of March.

https://carolina.codes

Post reply on HN