Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

River: A fast, robust job queue for Go and Postgres

brandur.org

Post reply on HN
Print view View original

41–50 of 112 posts

Re: River: A fast, robust job queue for Go and Postgres

#41

Post by eximius »

hipadev23 wrote: post #17
teraflop wrote: post #7

Earlier quoted context omitted.

It's not strange at all to me. The job is "transactional" in the sense that it depends on the transaction, and should be triggered iff the transaction commits. That doesn't mean it should run inside the transaction (especially since long-running transactions are terrible for performance). Passing around the job's data separately means that now you're storing two copies, which means you're creating a point where thing…

> should be triggered iff the transaction commits Agreed. Which is why the design doesn't make any sense. Because in the scenario presented they're starting a job during a transaction.

That part is somewhat poorly explained. That is a motivating example of why having your job queue system be separate from your system of record can be bad.

e.g.,

1. Application starts transaction 2. Application updates DB state (business details) 3. Application enqueues job in Redis 4. Redis jobworkers pick up job 5. Redis jobworkers error out 6. Application commits transaction

This motivates placing the jobworker state in the same transaction whereas non-DB based job queues have issues like this.

Re: River: A fast, robust job queue for Go and Postgres

#42

Post by tombh »

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?

Re: River: A fast, robust job queue for Go and Postgres

#43

Post by mosen »

tombh wrote: post #42
bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?

Looks like it’s underway:

> We're hard at work on more advanced features including a self-hosted web interface. Sign up to get updates on our progress.

Re: River: A fast, robust job queue for Go and Postgres

#44

Post by jbverschoor »

latchkey wrote: post #12
jbverschoor wrote: post #10

Earlier quoted context omitted.

>> Timeouts: for all HTTP Target task handlers the default timeout is 10 minutes, with a maximum of 30 minutes. Good luck with a long running batch.

If you're going to implement your own queue, you can make it run for however long you want. Again, I'm getting downvoted. The whole point of my comment isn't about using GCP Tasks, it is about what I would do if I was going to implement my own queue system like the author did. By the way, that 30 minute limitation can be worked around with checkpoints or breaking up the task into smaller chunks. Something that isn't…

Well you can't really.. If you're gonna use HTTP and expect a response, you're gonna be in for a fun ride. You'll have to go deal with timeout settings for:

  - http libraries
  - webservers
  - application servers
  - load balancers
  - reverse proxy servers
  - the cloud platform you're running on
  - waf
It might be alright for smaller "tasks", but not for "jobs".

Re: River: A fast, robust job queue for Go and Postgres

#45

Post by cloverich »

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

So excited y'all created this. Through a few job changes I've been exposed to the most popular background job systems in Rails, Python, JS, etc, and have been shocked at how under appreciated their limitations are relative to what you get out of the box with relational systems. Often I see a lot of DIY add-ons to help close the gaps, but its a lot of work and often still missing tons of edge cases and useful functionality. I always felt going the other way, starting w/ a relational db where many of those needs are free, would make more sense for most start-ups, internal tooling, and smaller scale businesses.

Thank you for this work, I look forward to taking it for a (real) test drive!

Re: River: A fast, robust job queue for Go and Postgres

#46

Post by bgentry »

tombh wrote: post #42
bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?

The UI isn't quite ready for outside consumption yet but it is being worked on. I would love to hear more about what you'd like to see in it if you want to share.

Re: River: A fast, robust job queue for Go and Postgres

#47

Post by latchkey »

jbverschoor wrote: post #44
latchkey wrote: post #12

Earlier quoted context omitted.

If you're going to implement your own queue, you can make it run for however long you want. Again, I'm getting downvoted. The whole point of my comment isn't about using GCP Tasks, it is about what I would do if I was going to implement my own queue system like the author did. By the way, that 30 minute limitation can be worked around with checkpoints or breaking up the task into smaller chunks. Something that isn't…

Well you can't really.. If you're gonna use HTTP and expect a response, you're gonna be in for a fun ride. You'll have to go deal with timeout settings for: - http libraries - webservers - application servers - load balancers - reverse proxy servers - the cloud platform you're running on - waf It might be alright for smaller "tasks", but not for "jobs".

Have you ever used Cloud Tasks?
Post reply on HN