River: A fast, robust job queue for Go and Postgres • Hacker News Board

Re: River: A fast, robust job queue for Go and Postgres

#61

Post by sorentwo » Mon, Nov 20, 2023, 8:09 PM UTC

The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo.

Starting with the project's tagline, "Robust job processing in Elixir", let's see what else:

  - The same job states, including the British spelling for `cancelled`
  - Snoozing and cancelling jobs inline
  - The prioritization system
  - Tracking where jobs were attempted in an attempted_by column
  - Storing a list of errors inline on the job
  - The same check constraints and the same compound indexes
  - Almost the entire table schema, really
  - Unique jobs with the exact same option names
  - Table-backed leadership election

Please give some credit where it's due.

[1]: https://github.com/sorentwo/oban

Re: River: A fast, robust job queue for Go and Postgres

#62

Post by gregwebs » Mon, Nov 20, 2023, 8:09 PM UTC

endorphine wrote: post #59

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

How does this compare to https://github.com/vgarvardt/gue ?

Or neoq. https://news.ycombinator.com/item?id=38352778

Re: River: A fast, robust job queue for Go and Postgres

#63

Post by bgentry » Mon, Nov 20, 2023, 8:10 PM UTC

JoshGlazebrook wrote: post #36

I didn't really see this feature, but I think another good one would be a way to schedule a future job that is not periodic. ie: "schedule job in 1 hr" to where it's either not enqueued or not available to be consumed until (at least) the schedule time.

You've found an underdocumented feature, but in fact River does already do what you're asking for! Check out `ScheduledAt` on the `InsertOpts`: https://pkg.go.dev/github.com/riverqueue/river#InsertOpts

I'll try to work this into the higher level docs website later today with an example :)

Re: River: A fast, robust job queue for Go and Postgres

#64

Post by csarva » Mon, Nov 20, 2023, 8:15 PM UTC

endorphine wrote: post #59

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

How does this compare to https://github.com/vgarvardt/gue ?

Not familiar with either project, but it seems gue is a fork of the authors previous project, https://github.com/bgentry/que-go

Re: River: A fast, robust job queue for Go and Postgres

#65

Post by radicalbyte » Mon, Nov 20, 2023, 8:25 PM UTC

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

I don't know why people even use libraries in those languages - assuming you stick to a database engine you understand well then the (main)-database-as-queue pattern is trivial to implement. Any time spent writing code is quickly won back by not having to debug weird edge cases, and sometimes you can highly optimize what you're doing (for example it becomes easy to migrate jobs which are data-dominated to the DB server which can cut processing time by 2-3 orders of magnitude).

It's particularly suited to use cases such background jobs, workflows or other operations which occur within your application and scales well enough for what 99.9999% of us will be doing.

Re: River: A fast, robust job queue for Go and Postgres

#66

Post by kamikaz1k » Mon, Nov 20, 2023, 8:32 PM UTC

sorentwo wrote: post #61

The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…

…how else do you spell cancelled? With one l? Wow, learning this on my keyboard as I type this…

Re: River: A fast, robust job queue for Go and Postgres

#67

Post by [deleted] » Mon, Nov 20, 2023, 8:38 PM UTC

bojanz wrote: post #58

This looks like a great effort and I am looking forward to trying it out. I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed pre…

[deleted]

Re: River: A fast, robust job queue for Go and Postgres

#68

Post by bgentry » Mon, Nov 20, 2023, 8:45 PM UTC

sorentwo wrote: post #61

The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…

Hi Parker, I'm genuinely sorry it comes across as though we lifted this stuff directly from Oban. I do mean it when I say that Oban has been a huge inspiration, particularly around its technical design and clean UX.

Some of what you've mentioned are cases where we surveyed a variety of our favorite job engines and concluded that we thought Oban's way was superior, whereas others we cycled through a few different implementations before ultimately apparently landing in a similar place. I'm not quite sure what to say on the spelling of "cancelled" though, I've always written it that way and can't help but read "canceled" like "concealed" in my head :)

As I think I mentioned when we first chatted years ago this has been a hobby interest of mine for many years so when a new database queue library pops up I tend to go see how it works. We've been in a bit of a mad dash trying to get this ready for release and didn't even think about crediting the projects that inspired us, but I'll sync with Brandur and make sure we can figure out the right way to do that.

I really appreciate you raising your concerns here and would love to talk further if you'd like. I just sent you an email to reconnect.

Re: River: A fast, robust job queue for Go and Postgres

#69

Post by bgentry » Mon, Nov 20, 2023, 8:56 PM UTC

dangoodmanUT wrote: post #57

bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

How do you look at models like temporal.io (service in front of DB) and go-workflows (direct to DB) in comparison? It seems like this is more a step back towards that traditional queue like asynq is, which is where the industry is leaving from to the model of temporal

I don't think these approaches are necessarily mutually exclusive. There are some great things that can be layered on top of the foundation we've built, including workflows. The best part about doing this is that you can maintain full transactionality within a single primary data store and not introduce another 3rd party or external service into your availability equation.

Re: River: A fast, robust job queue for Go and Postgres

#70

Post by bgentry » Mon, Nov 20, 2023, 8:59 PM UTC

csarva wrote: post #64

endorphine wrote: post #59

Earlier quoted context omitted.

How does this compare to https://github.com/vgarvardt/gue ?

Not familiar with either project, but it seems gue is a fork of the authors previous project, https://github.com/bgentry/que-go

Yes, there's a note in the readme to that effect although I don't think they bear much resemblance anymore. que-go was an experiment I hacked up on a plane ride 9 years ago and thought was worth sharing. I was never happy with its technical design: holding a transaction for the duration of a job severely limits available use cases and worsens bloat issues. It was also never something I intended to continue developing alongside my other priorities at the time and I should have made that clearer in the project's readme from the start.