Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

River: A fast, robust job queue for Go and Postgres

brandur.org

Post reply on HN
Print view View original

101–110 of 112 posts

Re: River: A fast, robust job queue for Go and Postgres

#101

Post by Rapzid »

qaq wrote: post #22
hipadev23 wrote: post #14

Earlier quoted context omitted.

> Wanting to offload heavy work to a background job is absolute as old of a best practice as exists in modern software engineering. Yes. I am intimately familiar with background jobs. In fact I've been using them long enough to know, without hesitation, that you don't use a relational database as your job queue.

Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.

Probably order of magnitude more or perhaps a multiple of that depending on the hardware and design.

In theory an append-only and/or HOT strategy leaning on Postgres just ripping through moderate sized in-mem lists could be incredibly fast. Design would be more complicated and perhaps use case dependent but I bet could be done.

Re: River: A fast, robust job queue for Go and Postgres

#102

Post by ddorian43 »

RedShift1 wrote: post #54
rockwotj wrote: post #31

Earlier quoted context omitted.

To work with each language's drivers

But it's the same thing every time. Turn autocommit off, run the SELECT, commit, repeat? Or am I missing something?

You just do the update with skip locked and finish the job until time runs out or update the job "liveness". (can be autocommit on & off)

Re: River: A fast, robust job queue for Go and Postgres

#103

Post by joker666 »

bgentry wrote: post #90
sorentwo wrote: post #55

> Work in a transaction has other benefits too. Postgres’ NOTIFY respects transactions, so the moment a job is ready to work a job queue can wake a worker to work it, bringing the mean delay before work happens down to the sub-millisecond level. Oban just went the opposite way, removing the use of database triggers for insert notifications and moving them into the application layer instead[1]. The prevalence of poole…

That makes a lot of sense, I've had the thought a few times that the NOTIFY overhead could get overwhelming in a high-throughput queue but haven't yet had an opportunity to verify this or experiment with a mechanism for reducing this overhead.

Also pgbouncer.

Re: River: A fast, robust job queue for Go and Postgres

#104

Post by qaq »

Rapzid wrote: post #101
qaq wrote: post #22

Earlier quoted context omitted.

Postgres based job queues work fine if you have say 10K transaction per second and jobs on average do not take significant time to complete (things will run fine on fairly modest instance). They also give guarantees that traditional job queues do not.

Probably order of magnitude more or perhaps a multiple of that depending on the hardware and design. In theory an append-only and/or HOT strategy leaning on Postgres just ripping through moderate sized in-mem lists could be incredibly fast. Design would be more complicated and perhaps use case dependent but I bet could be done.

Yep that's why I specifically mentioned "fairly modest instance" on reasonably fast box you can get magnitude more. You can partition the tasks table to reduce the number of rows skip locked has to run through to grab next task.

Re: River: A fast, robust job queue for Go and Postgres

#105

Post by bgentry »

justinclift wrote: post #95
bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

Does it do job completion notification? Along the lines of: _, err := river.Execute(context.Background(), j) // Enqueue the job, and wait for completion if err != nil { log.Fatalf("Unable to execute job: %s", err) } log.Printf("Job completed") Does that make sense?

It sounds like you're looking to be able to find out when a job has finished working, no matter which node it was run on. No, River does not have a mechanism for that today. It's definitely something we've talked about though.

Re: River: A fast, robust job queue for Go and Postgres

#106

Post by acaloiar »

gregwebs wrote: post #52

We are looking right now to use a stable PG job queue built in Go. We have found 2 already existing ones: * neoq: https://github.com/acaloiaro/neoq * gue: https://github.com/vgarvardt/gue Neoq is new and we found it to have some features (like scheduling tasks) that were attractive. The maintainer has also been responsive to fixing our bug reports and addressing our concerns as we try it out. Gue has been around for…

Hey thanks for the plug!

River's professional looking website makes me think there are more commercial ambitions behind River than neoq, but maybe I'm wrong. I do think they're very similar options, but neoq has no commercial ambitions. I simply set out to create a good, open source, queue-agnostic solution for Go users.

I'm actually switching its license to BSD or MIT this week to highlight that fact.

Re: River: A fast, robust job queue for Go and Postgres

#107

Post by AtlasBarfed »

radicalbyte wrote: post #65
bgentry wrote: post #21

Hi HN, I'm one of the authors of River along with Brandur. We've been working on this library for a few months and thought it was about time we get it out into the world. Transactional job queues have been a recurring theme throughout my career as a backend and distributed systems engineer at Heroku, Opendoor, and Mux. Despite the problems with non-transactional queues being well understood I keep encountering these…

I don't know why people even use libraries in those languages - assuming you stick to a database engine you understand well then the (main)-database-as-queue pattern is trivial to implement. Any time spent writing code is quickly won back by not having to debug weird edge cases, and sometimes you can highly optimize what you're doing (for example it becomes easy to migrate jobs which are data-dominated to the DB serv…

I thought the first rule of queues was never use a database as a queue.

Re: River: A fast, robust job queue for Go and Postgres

#108

Post by throwawaaarrgh »

whalesalad wrote: post #83
throwawaaarrgh wrote: post #81

Sorry if I'm late to the party, but has anyone told newbie developers that relational databases make very poor queues? We found that out like 15 years ago. (it's not about scalability, though that is a significant problem which this post hand-waves)

A lot has changed in 15 years.

Like?

Re: River: A fast, robust job queue for Go and Postgres

#109

Post by 22c »

AtlasBarfed wrote: post #107
radicalbyte wrote: post #65

Earlier quoted context omitted.

I don't know why people even use libraries in those languages - assuming you stick to a database engine you understand well then the (main)-database-as-queue pattern is trivial to implement. Any time spent writing code is quickly won back by not having to debug weird edge cases, and sometimes you can highly optimize what you're doing (for example it becomes easy to migrate jobs which are data-dominated to the DB serv…

I thought the first rule of queues was never use a database as a queue.

The "problem" is that a database as a queue doesn't infinitely scale, but nothing infinitely scales. If you're starting to reach the limits of Postgres-as-a-queue then you've either done something very wrong or very right to get to that point.

It's probably more correct to say that the engineering effort required to make a Postgres-as-a-queue scale horizontally is a lot more than the engineering effort required to make a dedicated queueing service scale horizontally. The trade-off is that you're typically going to have to start scaling horizontally much sooner with your dedicated queuing service than with a Postgres database.

The argument for Postgres-as-a-queue is that you might be able to get to market much quicker, which can be significantly more important than how well you can scale down the track.

Re: River: A fast, robust job queue for Go and Postgres

#110

Post by AtlasBarfed »

22c wrote: post #109
AtlasBarfed wrote: post #107

Earlier quoted context omitted.

I thought the first rule of queues was never use a database as a queue.

The "problem" is that a database as a queue doesn't infinitely scale, but nothing infinitely scales. If you're starting to reach the limits of Postgres-as-a-queue then you've either done something very wrong or very right to get to that point. It's probably more correct to say that the engineering effort required to make a Postgres-as-a-queue scale horizontally is a lot more than the engineering effort required to ma…

I thought Kafka could relatively infinitely scale
Post reply on HN