Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

River: A fast, robust job queue for Go and Postgres

brandur.org

Post reply on HN
Print view View original

21–30 of 112 posts

Re: River: A fast, robust job queue for Go and Postgres

#21

Post by bgentry »

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world.

Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these same problems. I wrote a bit about them here in our docs: https://riverqueue.com/docs/transactional-enqueueing

Ultimately I want to help engineers be able to focus their time on building a reliable product, not chasing down distributed systems edge cases. I think most people underestimate just how far you can get with this model—most systems will never outgrow the scaling constraints and the rest are generally better off not worrying about these problems until they truly need to.

Please check out the website and docs for more info. We have a lot more coming but first we want to iron out the API design with the community and get some feedback on what features people are most excited for. https://riverqueue.com/

Re: River: A fast, robust job queue for Go and Postgres

#22

Post by qaq »

hipadev23 wrote: post #14
brandur wrote: post #5

Earlier quoted context omitted.

Author here. Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. This is especially important for the kind of API and/or web development that a large number of people on this site are involved in. By offloading expensive work, you take that work out-of-band of the request that generated it, making that request faster and providing a far sup…

> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.

Re: River: A fast, robust job queue for Go and Postgres

#23

Post by bgentry »

victorbjorklund wrote: post #13

Looks great. For people wondering about wether postgres really is a good choice for a job queue I can recommend checking out Oban in Elixir that has been running in production for many years: https://github.com/sorentwo/oban Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps. https://getoban.pro/articles/one-million-jobs-a-minute-with-...

Oban is fantastic and has been a huge source of inspiration for us, showing what is possible in this space. In fact I think during my time at Distru we were one of Parker's first customers with Oban Web / Pro :)

We've also had a lot of experience with with other libraries like Que ( https://github.com/que-rb/que ) and Sidekiq (https://sidekiq.org/) which have certainly influenced us over the years.

Re: River: A fast, robust job queue for Go and Postgres

#24

Post by robertlagrant »

latchkey wrote: post #20
politician wrote: post #15

Earlier quoted context omitted.

Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack. https://brandur.org/job-drain https://news.ycombinator.com/item?id=15294722

"I'm into effective altruism and created the largest crypto exchange in the world. Cut me some slack." No, we don't operate like that. Call me out when I'm wrong technically, but don't tell me that because someone is some sort of celebrity that I should cut them some slack. Everything he pointed out is literally covered in the GCP Tasks documentation. https://cloud.google.com/tasks/docs/dual-overview https://cloud.go…

> No, we don't operate like that. Call me out when I'm wrong technically

You're being "called out" (ugh) incredibly politely mostly because you were being a bit rude; "tell me X without telling me" is just a bit unpleasant, and totally counterproductive.

> because someone is some sort of celebrity that I should cut them some slack.

No one mentioned a celebrity. You're not railing against the power of celebrity here; just a call for politeness.

> Everything he pointed out is literally covered in the GCP Tasks documentation.

Yes, e.g. as pitfalls.

Re: River: A fast, robust job queue for Go and Postgres

#25

Post by bgentry »

politician wrote: post #15
latchkey wrote: post #11

Earlier quoted context omitted.

> what if it fails, or what if you're not sure about whether it failed? This is covered in the GCP Tasks documentation. > There's definite scaling benefits to throwing tasks into Google's limitless compute power, but there's a lot of cases where a smaller, more correct queue is plenty of power, especially where Postgres is already the database of choice. My post was talking about what I would implement if I was doing…

Do you know that brandur's been writing about Postgres job queues since at least 2017? Cut him some slack. https://brandur.org/job-drain https://news.ycombinator.com/item?id=15294722

2015, even :) https://brandur.org/postgres-queues

Re: River: A fast, robust job queue for Go and Postgres

#26

Post by latchkey »

robertlagrant wrote: post #24
latchkey wrote: post #20

Earlier quoted context omitted.

"I'm into effective altruism and created the largest crypto exchange in the world. Cut me some slack." No, we don't operate like that. Call me out when I'm wrong technically, but don't tell me that because someone is some sort of celebrity that I should cut them some slack. Everything he pointed out is literally covered in the GCP Tasks documentation. https://cloud.google.com/tasks/docs/dual-overview https://cloud.go…

> No, we don't operate like that. Call me out when I'm wrong technically You're being "called out" (ugh) incredibly politely mostly because you were being a bit rude; "tell me X without telling me" is just a bit unpleasant, and totally counterproductive. > because someone is some sort of celebrity that I should cut them some slack. No one mentioned a celebrity. You're not railing against the power of celebrity here;…

Sure, updated my comment to be less rude.

Re: River: A fast, robust job queue for Go and Postgres

#29

Post by maherbeg »

hipadev23 wrote: post #2

What a strange design. If a job is dependent on an extant transaction then perhaps the job should run in the same code that initiated the transaction instead of a outside job queue? Also you pass the data a job needs to run as part of the job payload. Then you don't have the "data doesn't exist" issue.

I think you may be misunderstanding the design here. The transaction for initiating the job is only for queuing. The dequeue and execution of the job happens in a separate process.

The example on the home page makes this clear where a user is created and a job is created at the same time. This ensures that the job is queued up with the user creation. If any parts of that initial transaction fails, then the job queuing doesn't actually happen.

Re: River: A fast, robust job queue for Go and Postgres

#30

Post by bennyp101 »

Nice, I've been using graphile-worker [0] for a while now, and it handles our needs perfectly, so I can totally see why you want something in the go world.

Just skimming the docs, can you add a job directly via the DB? So a native trigger could add a job in? Or does it have to go via a client?

[0] https://worker.graphile.org/

Post reply on HN