Live data from Hacker News

Turning PostgreSQL into a queue serving 10k jobs per second (2013)

gist.github.com

11–20 of 146 posts

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#11
I have also found the lack of transactional guarantees in typical job queues to be very problematic.

One problem with using PostgreSQL in this way (using either advisory locks or LOCK FOR UPDATE) is that it requires you to keep an open connection to the database whilst the job is being worked on.

For a MySQL database, this would be just fine, but PostgreSQL uses a process-per-connection model which caps the number of active connections to the database to a relatively low number (on the order of 1000x fewer connections than a similarly sized MySQL instance) and tools like PgBouncer do nothing to help with this.

As a result, if your jobs take more than a few milliseconds to execute (let's say you make external HTTP requests as part of your job) this is not a good approach to take.

I use a similar approach which avoids this problem, but it only works because I have relatively low throughput requirements. I essentially implement in-database advisory locks using a separate table - before taking a job, workers create a row in the table, and the primary key of this table is used as a worker ID. Jobs are "taken" by assigning them a worker ID. Each row in the worker table has an expiry date, so if workers die, the corresponding row will be deleted and any linked jobs released back into the queue.

As well as transactional guarantees, using a database as a job queue gives you a lot of power over how jobs are executed: for example, our service for delivering webhooks has a separate queue per customer, and we can ensure that within a single queue jobs are processed strictly in order. Meanwhile, our service for search indexing supports different priority levels, so that newly created records are indexed with a higher priority.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#14
One of the interesting unforeseen downsides of RDBMS-based queues is explored here: https://brandur.org/postgres-queues

TDLR the lock time grows exponentially depending on the number of dead tuples in the table, which naturally grows as you use long running transactions.

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#15
The author of this post, Chris Hanks, created the Que queueing library for Ruby: https://github.com/que-rb/que

It’s changed significantly since this post as the 1.x betas use a very different structure which should actually be more efficient, use fewer Postgres connections, cause less lock contention, and cause less table bloat.

Not sure if the benchmarks have been run recently or not but I’m definitely curious how things stack up to this post from 6 years ago :)

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#16
post #3

How does this compare to Redis? Seems like Redis would handily beat it.

If your requirements don’t necessitate such scale, I can see Postgres being a viable alternative. I’d rather not introduce another dependency unless it’s absolutely needed.

I’ve never in my life worked on something where we didn’t keep having to upgrade our database resources

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#18

One of the interesting unforeseen downsides of RDBMS-based queues is explored here: https://brandur.org/postgres-queues TDLR the lock time grows exponentially depending on the number of dead tuples in the table, which naturally grows as you use long running transactions.

Is that fixed by the VACUUM process?

Re: Turning PostgreSQL into a queue serving 10k jobs per second (2013)

#19
post #12

Bad idea to use that because if the worker crash the event is lost, don't use PG for that.

Wait, what do you mean here? The job is a row in a database. You can crash as much as you want, the row is not going anywhere until you explicitly mark it as "worked" and delete it.

There are different performance constraints to this approach, but data integrity and robustness of it are unmatched, really.

Post reply on HN