River: A fast, robust job queue for Go and Postgres
1–10 of 112 posts
Re: River: A fast, robust job queue for Go and Postgres
#2What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue?
Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
Re: River: A fast, robust job queue for Go and Postgres
#3If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0].
It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution.
Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not suggesting that they use the GCP tasks product. That said, I would have just used GCP tasks too (assuming the usecase dictated it, fantastic and rock solid product.)
Re: River: A fast, robust job queue for Go and Postgres
#4What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
Re: River: A fast, robust job queue for Go and Postgres
#5What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
Author here.
Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering.
This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far superior user experience.
Example: User sign-up where you want to send a verification email. Talking to a foreign API like Mailgun might be a 100 ms to multisecond (worst case scenario) operation — why make the user wait on that? Instead, send it to the background, and give them a tight < 100 ms sign up experience that's so fast that for all intents and purposes, it feels instant.
Re: River: A fast, robust job queue for Go and Postgres
#6What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far sup…
GP isn’t taking umbrage with the concept of needing to offload work to a background process.
Re: River: A fast, robust job queue for Go and Postgres
#7What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance).
Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where things can get out of sync.
Re: River: A fast, robust job queue for Go and Postgres
#8What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.
I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data.
We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
Re: River: A fast, robust job queue for Go and Postgres
#9If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…
There's a lot to be said about the correctness benefits of a transactional model.
The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for. In practice, this leads to a lot of uncertainty around the edges, and operators having to reconcile things manually.
There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.
Re: River: A fast, robust job queue for Go and Postgres
#10If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…
>> Timeouts: for all HTTP Target task handlers the default timeout is 10 minutes, with a maximum of 30 minutes.
Good luck with a long running batch.