Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres LISTEN/NOTIFY does not scale

recall.ai

Post reply on HN
Print view View original

111–120 of 328 posts

Re: Postgres LISTEN/NOTIFY does not scale

#111

Post by [deleted] »

const_cast wrote: post #101
0xbadcafebee wrote: post #43

RBDMS are not designed for write-heavy applications, they are designed for read-heavy analysis. Also, an RDBMS is not a message queue or an RPC transport. I feel like somebody needs to write a book on system architecture for Gen Z that's just filled with memes. A funny cat pic telling people not to use the wrong tool will probably make more of an impact than an old fogey in a comment section wagging his finger.

People have been using RDBMS' for write-heavy workflows for forever. Some people even use stored procs or triggers for getting complicated write operations to work properly. Databases can do a lot of stuff, and if you're not hurting for DB performance it can be a good idea to just... do it in the database. The advantage is that, if the DB does it, you're much less likely to break things. Putting data constraints in a…

[deleted]

Re: Postgres LISTEN/NOTIFY does not scale

#112

Post by mind-blight »

qianli_cs wrote: post #89
leontrolski wrote: post #17

I'd be interested as to how dumb-ol' polling would compare here (the FOR UPDATE SKIP LOCKED method https://leontrolski.github.io/postgres-as-queue.html ). One day I will set up some benchmarks as this is the kind of thing people argue about a lot without much evidence either way. Wasn't aware of this AccessExclusiveLock behaviour - a reminder (and shameless plug 2) of how Postgres locks interact: https://leontrolski.…

My colleague did some internal benchmarking and found that LISTEN/NOTIFY performs well under low to moderate load, but doesn't scale well with a large number of listeners. Our findings were pretty consistent with this blog post. (Shameless plug [1]) I'm working on DBOS, where we implemented durable workflows and queues on top of Postgres. For queues, we use FOR UPDATE SKIP LOCKED for task dispatch, combined with expo…

Nice! I'm using DBOS and am a little active on the discord. I was just wondering how y'all handled this under the hood. Glad to hear I don't have to worry much about this issue

Re: Postgres LISTEN/NOTIFY does not scale

#113

Post by 0xbadcafebee »

hombre_fatal wrote: post #49
0xbadcafebee wrote: post #43

RBDMS are not designed for write-heavy applications, they are designed for read-heavy analysis. Also, an RDBMS is not a message queue or an RPC transport. I feel like somebody needs to write a book on system architecture for Gen Z that's just filled with memes. A funny cat pic telling people not to use the wrong tool will probably make more of an impact than an old fogey in a comment section wagging his finger.

But those rules of thumb aren't true. People use Postgres for job queues and write-heavy applications. You'd have to at least accompany your memes with empirics. What is write-heavy? A number you might hit if your startup succeeds with thousands of concurrent users on your v1 naive implementation? Else you just get another repeat of everyone cargo-culting Mongo because they heard that Postgres wasn't web scale for th…

There are lots of ways to empirically tell what solutions are right for what applications. The simplest is using basic computer science like applying big-O notation, or using something designed as a message queue to do message queueing, etc. Slightly more complicated are simple benchmarks with immutable infrastructure.

Re: Postgres LISTEN/NOTIFY does not scale

#114

Post by 0xbadcafebee »

const_cast wrote: post #101
0xbadcafebee wrote: post #43

RBDMS are not designed for write-heavy applications, they are designed for read-heavy analysis. Also, an RDBMS is not a message queue or an RPC transport. I feel like somebody needs to write a book on system architecture for Gen Z that's just filled with memes. A funny cat pic telling people not to use the wrong tool will probably make more of an impact than an old fogey in a comment section wagging his finger.

People have been using RDBMS' for write-heavy workflows for forever. Some people even use stored procs or triggers for getting complicated write operations to work properly. Databases can do a lot of stuff, and if you're not hurting for DB performance it can be a good idea to just... do it in the database. The advantage is that, if the DB does it, you're much less likely to break things. Putting data constraints in a…

That is the same logic that led every failed design I've seen in my career to take months (if not years) and tons of money to fix. "YOLO engineering" is simple at first and a huge pain in the ass later. Whereas actually correct engineering is slightly painful at first and saves your ass later.

The people who design it walk away after a few years, so they don't give a crap what happens. The rest of us have to struggle to support or try to replace whatever the lumbering monstrosity is.

Re: Postgres LISTEN/NOTIFY does not scale

#115

Post by chrnola »

j16sdiz wrote: post #94
kristianc wrote: post #84

Earlier quoted context omitted.

Sounds like a deliberate attempt to avoid spinning up Redis, Kafka, or an outbox system early on.. and then underestimated how quickly their scale would make it blow up. Story as old as time.

Kafka head of line blocking sucks.

Guaranteeing order has its tradeoffs.

There is work happening currently to make Kafka behave more like a queue: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

Re: Postgres LISTEN/NOTIFY does not scale

#116

Post by tecleandor »

sorentwo wrote: post #51
alberth wrote: post #44

Earlier quoted context omitted.

I didn’t realize Oban didn’t use Mnesia (Erlang built-in).

Very very few applications use mnsesia. There’s absolutely no way I would recommend it over Postgres.

I think RabbitMQ still uses by default for its metadata storage. Is it problematic?

Re: Postgres LISTEN/NOTIFY does not scale

#117

Post by TheTaytay »

callamdelaney wrote: post #69

My kneejerk reaction to the headline is ‘why would it?’. It’s unsurprising to me that an AI company appears to have chosen exactly the wrong tool for the job.

Because documentation doesn’t warn about this well-loved feature effectively ruins the ability to perform parallel writes, and because everything else in Postgres scales well.

I think it’s a reasonable assumption. Based on the second half of your comment, you clearly don’t think highly of “AI companies,” but I think that’s a separate issue.

Re: Postgres LISTEN/NOTIFY does not scale

#118

Post by cheesekunator »

Groxx wrote: post #93
parthdesai wrote: post #61

Earlier quoted context omitted.

What happens when transaction succeeds but the execution of NOTIFY fails if it's outside of transaction, in it's own separate connection?

The same thing that happens if the notified process dies suddenly. If you're not handling that, then whatever you're doing is unreliable either way.

98% of developers can't see it

Re: Postgres LISTEN/NOTIFY does not scale

#119

Post by spooneybarger »

asg0451 wrote: post #102
sorentwo wrote: post #51

Earlier quoted context omitted.

Very very few applications use mnsesia. There’s absolutely no way I would recommend it over Postgres.

can you explain why?

Mnesia along with clustering was a recipe for split brain disasters a few years ago I assume that hasn't been addressed.

Re: Postgres LISTEN/NOTIFY does not scale

#120

Post by valenterry »

saltcured wrote: post #65
parthdesai wrote: post #61

Earlier quoted context omitted.

What happens when transaction succeeds but the execution of NOTIFY fails if it's outside of transaction, in it's own separate connection?

For reliability, you can make the recipient poll the table(s) of record for relevant state and use the out-of-band notification channel as a latency-reducer. So, the poller is eventually consistent at some configured polling interval, but opportunistically can respond much sooner when told to check again ahead of the next scheduled poll time. In my experience, this means you make sure the polling solution is complete…

> you can make the recipient poll the table(s) of record for relevant state

That is tricky due to transactions and visibility. How do you write the poller to not miss events that were written by a long/blocked transaction? You'd have to set the poller scan to a long time (e.g. "process events that were written since now minus 5minutes") and then make sure transactions are cancelled hard before those 5minutes.

Post reply on HN