Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)
River: A fast, robust job queue for Go and Postgres
81–90 of 112 posts
Re: River: A fast, robust job queue for Go and Postgres
#82Earlier quoted context omitted.
> No, we don't operate like that. Call me out when I'm wrong technically You're being "called out" (ugh) incredibly politely mostly because you were being a bit rude; "tell me X without telling me" is just a bit unpleasant, and totally counterproductive. > because someone is some sort of celebrity that I should cut them some slack. No one mentioned a celebrity. You're not railing against the power of celebrity here;…
Sure, updated my comment to be less rude.
Appreciated
Re: River: A fast, robust job queue for Go and Postgres
#83Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)
A lot has changed in 15 years.
Re: River: A fast, robust job queue for Go and Postgres
#84Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…
How do you look at models like temporal.io (service in front of DB) and go-workflows (direct to DB) in comparison? It seems like this is more a step back towards that traditional queue like asynq is, which is where the industry is leaving from to the model of temporal
Personally I've found Temporal very limited as a queue: no FIFO ordering (!), no priorities, no routing, etc. It's also very complex when you just want async jobs, and more specialized than a DB or message broker, which can be used for many other things. I think there's a place for both.
Re: River: A fast, robust job queue for Go and Postgres
#85If I was going to do my own Job Queue, I'd implement it more like the GCP Tasks [0]. It is such a better model for the majority of queues. All you're doing is storing a message, hitting an HTTP endpoint and deleting the message on success. This makes it so much easier to scale, reason, and test task execution. Update: since multiple people seem confused. I'm talking about the implementation of a job queue system, not…
that's pretty fundamentally different, no? One requires you to build a distributed system with >1 components leveraging GCP tasks APIs. The second is just a library do some book keeping inside your main datastore.
Re: River: A fast, robust job queue for Go and Postgres
#86The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…
As if Oban invented anything, like queues on RDMS is not a new concept at all. Oban is 4 years old, do you know how many queues baked by DB were created in the last 10years?
I don't see Sidekiq credited on the main page of Oban.
Re: River: A fast, robust job queue for Go and Postgres
#87The number of features lifted directly from Oban[1] is astounding, considering there isn't any attribution in the announcement post or the repo. Starting with the project's tagline, "Robust job processing in Elixir", let's see what else: - The same job states, including the British spelling for `cancelled` - Snoozing and cancelling jobs inline - The prioritization system - Tracking where jobs were attempted in an att…
As if Oban invented anything, like queues on RDMS is not a new concept at all. Oban is 4 years old, do you know how many queues baked by DB were created in the last 10years? I don't see Sidekiq credited on the main page of Oban.
I’d argue strongly that Oban did invent things, including parts of the underlying structure used in River, and the authors agree that it was a heavy influence.
While there is no overlap in technology or structure with Sidekiq, the original Oban announcement on the ElixirForum mentions it along with all of the direct influences:
https://elixirforum.com/t/oban-reliable-and-observable-job-p...
Re: River: A fast, robust job queue for Go and Postgres
#88Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)
A lot has changed in 15 years.
[deleted]
Re: River: A fast, robust job queue for Go and Postgres
#89Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)
A lot has changed in 15 years.
[deleted]
Re: River: A fast, robust job queue for Go and Postgres
#90> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level. Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poole…
That makes a lot of sense, I've had the thought a few times that the NOTIFY overhead could get overwhelming in a high-throughput queue but haven't yet had an opportunity to verify this or experiment with a mechanism for reducing this overhead.