All these job queue implementations do the same thing right, SELECT ... FOR UPDATE SKIP LOCKED? Why does every programming language need its own variant?
To work with each language's drivers
River: A fast, robust job queue for Go and Postgres
31–40 of 112 posts
Re: River: A fast, robust job queue for Go and Postgres
#32I love PG job queues! They’re surprisingly easy to implement in plain SQL: [1] https://taylor.town/pg-task The nice thing about this implementation is that you can query within the same transaction window
Agreed. Shortwave [1] is built completely on this, but with the added layer of having a leasing system that is per user on top of the tasks. So you only need to `SKIP LOCKED` to grab a lease, then you can grab as many tasks as you want and process them in bulk. It allows higher throughput of tasks, and also was required for the use case as the leases where tied to a user and tasks for a single user must be processed in order.
Re: River: A fast, robust job queue for Go and Postgres
#33Looks cool and thanks for sharing. Founder of windmill.dev, an open-source, extremely fast workflow engine to run jobs in ts,py,gosh whose most important piece, the queue, is also just rust + postgresql (and mostly the FOR UPDATE SKIP LOCKED).
I'd be curious to compare performances once you guys are comfortable with that, we do them openly and everyday on: https://github.com/windmill-labs/windmill/tree/benchmarks
I wasn't aware of the skip B-tree splits and the REINDEX CONCURRENTLY tricks. But curious what do you index in your jobs that use those. We mostly rely on the tag/queue_name (which has a small cardinality), scheduled_for, and running boolean which don't seem good fit for b-trees.
Re: River: A fast, robust job queue for Go and Postgres
#34If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…
There's a lot to be said about the correctness benefits of a transactional model. The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for.…
HTTP APIs are ideal for message queues with Postgres.
The request to get a message returns a token that identifies this receive.
You use that token to delete the message when you are done.
Jobs that don’t succeed after N retries get marked as dead and go into the dead letter list.
This the way AWS SQS works, it’s tried and true.
Re: River: A fast, robust job queue for Go and Postgres
#35Earlier quoted context omitted.
It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…
> should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.
Maybe it’s not designed for that or all use cases and that can make sense.
Personally, I need long running jobs.
Re: River: A fast, robust job queue for Go and Postgres
#36I didn't really see this feature, but I think another good one would be a way to schedule a future job that is not periodic. ie: "schedule job in 1 hr" to where it's either not enqueued or not available to be consumed until (at least) the schedule time.
Re: River: A fast, robust job queue for Go and Postgres
#37The thing is there are now vast numbers of queue solutions.
What’s the goal for the project? Is it to be commercial? If so you face massive headwind because it’s so incredibly easy to implement a queue now.
Re: River: A fast, robust job queue for Go and Postgres
#38Earlier quoted context omitted.
It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…
> should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.
I don't understand what you mean. The job is "created" as part of the transaction, so it only becomes visible (and hence eligible to be executed) when the transaction commits.
Re: River: A fast, robust job queue for Go and Postgres
#39Earlier quoted context omitted.
It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…
> should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.
The job is queued as part of the transaction. It is executed by a worker outside the scope of the transaction.
Re: River: A fast, robust job queue for Go and Postgres
#40Earlier quoted context omitted.
Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far sup…
> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.
> I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.
I'm also very familiar with jobs and I have used the usual tools like Redis and RMQ, but I wouldn't make a blanket statement like that. There are people using RDBS as queues in prod so we have some counter-examples. I wouldn't mind at all to get rid of another system (not just one server but the cluster of RMQ/Redis you need for HA). If there's a big risk in using pg as backend for a task queue, I'm all ears.