Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

111–120 of 369 posts

Re: Choose Postgres queue technology

#111
post #110

Earlier quoted context omitted.

I recently published a manifesto and code snippets for exactly this in Postgres! delete from task where task_id in ( select task_id from task order by random() -- use tablesample for better performance for update skip locked limit 1 ) returning task_id, task_type, params::jsonb as params [1] https://taylor.town/pg-task

Presumably it's okay that this loses work if your task runner has an error?

From the linked article

> The task row will not be deleted if sendEmail fails. The PG transaction will be rolled back. The row and sendEmail will be retried.

Re: Choose Postgres queue technology

#112
post #110

Earlier quoted context omitted.

I recently published a manifesto and code snippets for exactly this in Postgres! delete from task where task_id in ( select task_id from task order by random() -- use tablesample for better performance for update skip locked limit 1 ) returning task_id, task_type, params::jsonb as params [1] https://taylor.town/pg-task

Presumably it's okay that this loses work if your task runner has an error?

If you read my guide, you’ll see that I embed it in a transaction that doesn’t COMMIT until the companion code is complete :)

For example, I run the above query to grab a queued email, send it using mailgun, then COMMIT. Nothing is changed in the DB unless the email is sent.

Re: Choose Postgres queue technology

#113
post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Anything but RabbitMQ.

Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

Re: Choose Postgres queue technology

#115
post #82

We used postgres for some of our queues back when we were at ~10 msg/s. It scaled quite a bit, but, honestly, setting up SQS or some other queue stack in AWS, GCP, or Azure is so simple and purpose built for the task (with DL queues and the like built in), I don’t know why you wouldn’t just go that route and not have to worry about that system shitting the bed and affecting the rest of the DB’s health. It seems fooli…

What I've settled on is "store most job state in the DB, use task queues just to poke workers into working on the jobs". Storing the job state in the DB means you can query state nicely. It's not going to exactly show the state of things but it's helpful for working through a live incident (especially when most job queues just delete records as work is processed). And if you make all the background tasks idempotent a…

Ultimately you have to figure out the separation of concerns of the job state and other core state. Ranging from “all state stored in message and will never become out of sync” to “no state stored in message and will never become out of sync”. In between you have “some state stored in db and some in message” and what I’ve found to be useful is keeping stuff in the db that needs to have high end state integrity (or as you said just making sure jobs are cancellable/idempotent).

Tangible example:

We have a video transcoder queue. The state of the video model in our db can change as the video is being finalized in various ways. The transcoder generates thumbnails and assets from the video and also updates its state in the db. So we store job information in the message about what thumbnails we want to generate and the video ID but nothing else. This allows us to look up the video row, see if the same media was already transcoded from the video (and cancel the job), and, if not, run the job and update the video row.

Also (and I know you’re not saying this), but I’ve never understood the argument that keeping queues in Postgres leads to higher data integrity via transaction guarantees. The job is still running on another process outside of the db. The only time this could be true is if the job itself mostly updates state in the db, in which case it’s the small minority of queued workloads (with the majority needing to do non-db compute work).

Re: Choose Postgres queue technology

#116
post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Anything but RabbitMQ.

Bad experiences? If so, was it the tech itself or surrounding stuff? (aka maybe docs, maybe community, etc)

Re: Choose Postgres queue technology

#117

Earlier quoted context omitted.

I've done even simpler without locks (as no transaction logic), where I select a row, and then try to update a field about it being taken. If 1 row is affected, it's mine. If 0, someone else did it before me and I select a new row. I've used this for tasks at big organizations without issue. No need for any special deployments or new infra. Just spin up a few worker threads in your app. Perhaps a thread to reset aban…

PSA: This is a read-modify-write pattern, thus it is not safe under concurrency unless a transaction isolation level of SERIALIZABLE is specified, or some locking mechanism is used (select for update etc).

This should be safe under SI (other than the ABA issue, which isn't even fixed with serializable). The update forces a W-W conflict, which is sufficient to make the behavior serializable under SI (and therefore, I think but am not sure, PG's RR level too).

Re: Choose Postgres queue technology

#118
post #28

Few things. 1. The main downside to using PostgreSQL as a pub/sub bus with LISTEN/NOTIFY is that LISTEN is a session feature, making it incompatible with statement level connection pooling. 2. If you are going to do this use advisory locks [0]. Other forms of explicit locking put more pressure on the database while advisory locks are deliberately very lightweight. My favorite example implementation is que [1] which i…

One reason that makes me dislike NOTIFY/LISTEN is that issues with it are hard to diagnose.

Recently I had to stop using it because after a while all NOTIFY/LISTENS would stop working, and only a database restart would fix the issue https://dba.stackexchange.com/questions/325104/error-could-n...

Re: Choose Postgres queue technology

#119
post #103
post #76

Earlier quoted context omitted.

Parallel or not, the order is of importance in any queue system.

You have no ordering guarantees, so how can order be important? If 4 work items are scheduled on 4 independent workers, you have no guarantee which will start first or finish first.

The order matters in the sense that the 5th jobs should not be atempted before those 4.

Re: Choose Postgres queue technology

#120
I've been thinking a lot lately about how much of tech, and life more broadly, is ruined by the pursuit of scale.

Taking on problems you don't (and will never) have because some vanishingly small minority has experienced them is nuts. Over-engineering is as incorrect as under-engineering. The correctly sized, correctly complicated answer is what we're after.

Post reply on HN