Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

River: A fast, robust job queue for Go and Postgres

brandur.org

Post reply on HN
Print view View original

11–20 of 112 posts

Re: River: A fast, robust job queue for Go and Postgres

#11

Post by latchkey »

brandur wrote: post #9
latchkey wrote: post #3

If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…

There's a lot to be said about the correctness benefits of a transactional model. The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for.…

> what if it fails, or what if you're not sure about whether it failed?

This is covered in the GCP Tasks documentation.

> There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice.

My post was talking about what I would implement if I was doing my own queue, as the authors were. Not about using GCP Tasks.

Re: River: A fast, robust job queue for Go and Postgres

#12

Post by latchkey »

jbverschoor wrote: post #10
latchkey wrote: post #3

If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…

>> Timeouts: for all HTTP Target task handlers the default timeout is 10 minutes, with a maximum of 30 minutes. Good luck with a long running batch.

If you're going to implement your own queue, you can make it run for however long you want.

Again, I'm getting downvoted. The whole point of my comment isn't about using GCP Tasks, it is about what I would do if I was going to implement my own queue system like the author did.

By the way, that 30 minute limitation can be worked around with checkpoints or breaking up the task into smaller chunks. Something that isn't a bad idea to do anyway. I've seen long running tasks cause all sorts of downstream problems when they fail and then take forever to run again.

Re: River: A fast, robust job queue for Go and Postgres

#13

Post by victorbjorklund »

Looks great. For people wondering about wether postgres really is a good choice for a job queue I can recommend checking out Oban in Elixir that has been running in production for many years: https://github.com/sorentwo/oban

Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps.

https://getoban.pro/articles/one-million-jobs-a-minute-with-...

Re: River: A fast, robust job queue for Go and Postgres

#14

Post by hipadev23 »

brandur wrote: post #5
hipadev23 wrote: post #2

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far sup…

> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering.

Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

Re: River: A fast, robust job queue for Go and Postgres

#15

Post by politician »

latchkey wrote: post #11
brandur wrote: post #9

Earlier quoted context omitted.

There's a lot to be said about the correctness benefits of a transactional model. The trouble with hitting an HTTP API to queue a task is: what if it fails, or what if you're not sure about whether it failed? You can continue to retry in-band (although there's a definite latency disadvantage to doing so), but say you eventually give up, you can't be sure that no jobs were queued which you didn't get a proper ack for.…

> what if it fails, or what if you're not sure about whether it failed? This is covered in the GCP Tasks documentation. > There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice. My post was talking about what I would implement if I was doing…

Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack.

https://brandur.org/job-drain

https://news.ycombinator.com/item?id=15294722

Re: River: A fast, robust job queue for Go and Postgres

#16

Post by [deleted] »

teraflop wrote: post #7
hipadev23 wrote: post #2

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…

[deleted]

Re: River: A fast, robust job queue for Go and Postgres

#17

Post by hipadev23 »

teraflop wrote: post #7
hipadev23 wrote: post #2

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…

> should be triggered iff the transaction commits

Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.

Re: River: A fast, robust job queue for Go and Postgres

#18

Post by qaq »

hipadev23 wrote: post #2

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

Job is not dependent on extant transaction. The bookkeeping of job state runs in the same transaction as your domain state manipulation so you will never get into situation where job domain mutation commited but job state failed to update to complete.

Re: River: A fast, robust job queue for Go and Postgres

#19

Post by toolz »

hipadev23 wrote: post #14
brandur wrote: post #5

Earlier quoted context omitted.

Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far sup…

> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

as far as I'm aware the most popular job queue library in elixir depends on postgres and has performance characteristics that cover the vast majority of background processing needs I've come across.

I wonder maybe if you've limited yourself by assuming relational DBs only have features for relational data. That isn't the case now and really hasn't been the case for quite some time now.

Re: River: A fast, robust job queue for Go and Postgres

#20

Post by latchkey »

politician wrote: post #15
latchkey wrote: post #11

Earlier quoted context omitted.

> what if it fails, or what if you're not sure about whether it failed? This is covered in the GCP Tasks documentation. > There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice. My post was talking about what I would implement if I was doing…

Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack. https://brandur.org/job-drain https://news.ycombinator.com/item?id=15294722

"I'm into effective altruism and created the largest crypto exchange in the world. Cut me some slack."

No, we don't operate like that. Call me out when I'm wrong technically, but don't tell me that because someone is some sort of celebrity that I should cut them some slack.

Everything he pointed out is literally covered in the GCP Tasks documentation.

https://cloud.google.com/tasks/docs/dual-overview

https://cloud.google.com/tasks/docs/common-pitfalls

Post reply on HN