If you are on Kafka already, there is an alternative to schedule a job without PG [0] [0] https://www.wgtwo.com/blog/kafka-timers/
[deleted]
River: A fast, robust job queue for Go and Postgres
51–60 of 112 posts
Re: River: A fast, robust job queue for Go and Postgres
#52We are looking right now to use a stable PG job queue built in Go.
We have found 2 already existing ones:
* neoq: https://github.com/acaloiaro/neoq
* gue: https://github.com/vgarvardt/gue
Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out.
Gue has been around for a while and is probably serving its users well.
Looking forward to trying out River now. I do wonder if neoq and river might be better off joining forces.
Re: River: A fast, robust job queue for Go and Postgres
#53Looks great. For people wondering about wether postgres really is a good choice for a job queue I can recommend checking out Oban in Elixir that has been running in production for many years: https://github.com/sorentwo/oban Benchmark: peaks at around 17,699 jobs/sec for one queue on one node. Probably covers most apps. https://getoban.pro/articles/one-million-jobs-a-minute-with-...
Oban is fantastic and has been a huge source of inspiration for us, showing what is possible in this space. In fact I think during my time at Distru we were one of Parker's first customers with Oban Web / Pro :) We've also had a lot of experience with with other libraries like Que ( https://github.com/que-rb/que ) and Sidekiq ( https://sidekiq.org/ ) which have certainly influenced us over the years.
The very first paying Pro customer, as a matter of fact =)
You said back then that you planned on pursuing a Go client; now, four years later, here we are. River looks excellent, and the blog post does a fantastic job explaining all the benefits of job queues in Postgres.
Re: River: A fast, robust job queue for Go and Postgres
#54All these job queue implementations do the same thing right, SELECT ... FOR UPDATE SKIP LOCKED? Why does every programming language need its own variant?
To work with each language's drivers
But it's the same thing every time. Turn autocommit off, run the SELECT, commit, repeat? Or am I missing something?
Re: River: A fast, robust job queue for Go and Postgres
#55> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level.
Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poolers like pgbouncer, which prevent NOTIFY ever triggering, and the extra db load of trigger handling wasn't worth it.
[1]: https://github.com/sorentwo/oban/commit/7688651446a76d766f39...
Re: River: A fast, robust job queue for Go and Postgres
#56Earlier quoted context omitted.
I agree. This design is incredibly strange, and seems to throw away basically all distributed systems knowledge. I'm glad folks are playing with different ideas, but this one seems off.
No, this is a fairly common pattern called having an 'outbox' where the emission/enquing of your event/message/job is tied to the transaction completion of the relevant domain data. We use this to ensure Kafka events are only emitted when a process succeeds, this is very similar.
So when the the business data transaction commit a notify event is raised and a job row is inserted. Out of bound job broker listens to a notify event of the job-table or polls the table skipping rows and takes work for processing?
Re: River: A fast, robust job queue for Go and Postgres
#57Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…
How do you look at models like temporal.io (service in front of DB) and go-workflows (direct to DB) in comparison? It seems like this is more a step back towards that traditional queue like asynq is, which is where the industry is leaving from to the model of temporal
Re: River: A fast, robust job queue for Go and Postgres
#58This looks like a great effort and I am looking forward to trying it out.
I am a bit confused by the choice of the LGPL 3.0 license. It requires one to dynamically link the library to avoid GPL's virality, but in a language like Go that statically links everything, it becomes impossible to satisfy the requirements of the license, unless we ignore what it says and focus just on its spirit. I see that was discussed previously by the community in posts such as these [1][2][3]
I am assuming that bgentry and brandur have strong thoughts on the topic since they avoided the default Go license choice of BSD/MIT, so I'd love to hear more.
[1] https://www.makeworld.space/2021/01/lgpl_go.html [2] https://golang-nuts.narkive.com/41XkIlzJ/go-lgpl-and-static-... [3] https://softwareengineering.stackexchange.com/questions/1790...
Re: River: A fast, robust job queue for Go and Postgres
#59Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…
How does this compare to https://github.com/vgarvardt/gue?
Re: River: A fast, robust job queue for Go and Postgres
#60Earlier quoted context omitted.
At the bottom of the page on riverqueue.com it appears there's a screenshot of a UI. But I can't seem to find any docs about it. Am I missing something or is it just not available yet?
The UI isn't quite ready for outside consumption yet but it is being worked on. I would love to hear more about what you'd like to see in it if you want to share.
An Airflow for Gophers?