Making Postgres queues scale
dbos.dev
Making Postgres queues scale
1–10 of 36 posts
Re: Making Postgres queues scale
#2> The conventional wisdom around Postgres-backed queues is that they don't scale.
That might have been the case 10 years ago. In the past years there have been many Postgres powered queueing systems and even Rails switched to Postgres powered queues by default (SolidQueue) more than 3 years ago.
Re: Making Postgres queues scale
#3They certainly do, and I don't think it's a controversial take at this point.
Shameless link to an older article about throughput with Oban (https://oban.pro/articles/one-million-jobs-a-minute-with-oba...), and in follow-up research we've sustained 12k/s with a p99 under ~100ms.
Re: Making Postgres queues scale
#4Reading Postgres queuing posts always seem like deja vu. People love to write about them
Re: Making Postgres queues scale
#5Reading Postgres queuing posts always seem like deja vu. People love to write about them
These days, most of them are from the DBOS folks. :)
Re: Making Postgres queues scale
#6[flagged]
Re: Making Postgres queues scale
#7> The conventional wisdom around Postgres-backed queues is that they don't scale. That might have been the case 10 years ago. In the past years there have been many Postgres powered queueing systems and even Rails switched to Postgres powered queues by default (SolidQueue) more than 3 years ago.
I see a lot of back and forth about postgres' suitability as a queuing system. I wonder if there's a couple of separable problems here. Postgres backed queues - even very scalable ones - might work well for background jobs in a monolithic app backed by a single DB. Things like backgrounding sending an email etc.
But usually when I reach for a queuing system, it's because I want to decouple a part of the architecture. And in that case, it's probably better to use a dedicated queueing system instead of postgres.
I wonder if the two cases are conflated in a lot of online discussion.
Re: Making Postgres queues scale
#8Re: Making Postgres queues scale
#9> The conventional wisdom around Postgres-backed queues is that they don't scale. That might have been the case 10 years ago. In the past years there have been many Postgres powered queueing systems and even Rails switched to Postgres powered queues by default (SolidQueue) more than 3 years ago.
I see a lot of back and forth about postgres' suitability as a queuing system. I wonder if there's a couple of separable problems here. Postgres backed queues - even very scalable ones - might work well for background jobs in a monolithic app backed by a single DB. Things like backgrounding sending an email etc. But usually when I reach for a queuing system, it's because I want to decouple a part of the architecture.…
We are very heavily using Postgres as a queuing system in production for many years already, not just some quiet background tasks but with millions of tasks in the queue at any given moment. I have yet so see any issues with that so I'm always a bit suspicious when people say they had to reach for something else unless you are at a crazy scale.
I know it's very hard to compare workloads, but famous recent example: https://openai.com/index/scaling-postgresql/
> It may sound surprising that a single-primary architecture can meet the demands of OpenAI’s scale; however, making this work in practice isn’t simple.
Re: Making Postgres queues scale
#10> The conventional wisdom around Postgres-backed queues is that they don't scale. That might have been the case 10 years ago. In the past years there have been many Postgres powered queueing systems and even Rails switched to Postgres powered queues by default (SolidQueue) more than 3 years ago.
I have the impression that SolidQueue uses whatever your rails app is using by default. That can be SQLite, PostgreSQL, MySQL, etc.
So I don't think they changed to PostgreSQL per se.