Any suggestions for something like this for dotnet?
Updlock rowlock readpast should do the trick
Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
31–40 of 140 posts
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#32You might also want to look at River (https://github.com/riverqueue/river) for inspiration as they support scheduled jobs, etc.
From an end-user perspective, they also have a UI which is nice to have for debugging.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#33We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#34How does LISTEN/NOTIFY compare to using select for update skip locked? I thought listen/notify can lose queue items when the process crashes? Is that true? Do you need to code for those cases in some manner?
There are two things:
1. Signaling
2. Messaging
In some systems, those are, effectively, the same. A consumer listens, and the signal is the message. If the consumer process crashes, the message returns to the queue and gets processed when the consumer comes back online.
If the signal and messaging are separated, as in Postgres, where LISTEN/NOTIFY is the signal, and the skip locked query is the message pull, the consumer process would need to do some combination of polling and listening.
In the consumer, that could essentially be a loop that’s just doing the skip locked query on startup, then dropping into a LISTEN query only once there are no messages present in the queue. Then the LISTEN/NOTIFY is just signaling to tell the consumer to check for new messages.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#35there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
I mean I love postgres like the next guy. And I like simple solutions as long as they work.
I just wonder if this is truly simpler than using a redis or rabbitmq queue if you need Queues. If you're already using a cloud provider sqs is quite trivial as well.
I guess if you already have postgres and don't want to use the cloud provider's solution. You can use this to avoid hosting another piece of infra.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#36there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
I think for me the problem with every single new PG queue is that it seems like everyone and their mother thinks they need to reinvent this specific wheel for some reason and the flavor of the day doesn’t often bring much new to the space. Probably because it's
1. Pretty easy to understand and grok the problem space
2. Scratching the programmer itch of wanting something super generic that you can reuse all over the place
3. Doable with a modest effort over a reasonable scope of time
4. Built on rock solid internals (Postgres) with specific guarantees that you can lean on
Here's 7 of them just right quick:
- https://github.com/timgit/pg-boss
- https://github.com/queueclassic/queue_classic
- https://github.com/florentx/pgqueue
- https://github.com/mbreit/pg_jobs
- https://github.com/graphile/worker
- https://github.com/que-rb/que
Probably could easily find more by searching, I only spent about 5 minutes looking and grabbing the first ones I found.
I'm all for doing this kind of thing as an academic exercise, because it's a great way to learn about this problem space. But at this point if you're reinventing the Postgres job queue wheel and sharing it to this technical audience you need to probably also include why your wheel is particularly interesting if you want to grab my attention.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#37You might also want to look at River ( https://github.com/riverqueue/river ) for inspiration as they support scheduled jobs, etc. From an end-user perspective, they also have a UI which is nice to have for debugging.
I’ve been using river for some low volume stuff. I love that I can add a job to the queue in the same db transaction that handle the synchronous changes.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#38there seems to be a big hype to adapt pg into any infra. I love PG but this seems not be right thing.
I mean I love postgres like the next guy. And I like simple solutions as long as they work. I just wonder if this is truly simpler than using a redis or rabbitmq queue if you need Queues. If you're already using a cloud provider sqs is quite trivial as well. I guess if you already have postgres and don't want to use the cloud provider's solution. You can use this to avoid hosting another piece of infra.
db-based gives you the ability to query against your queues, if you use case needs it. Other options tend to dispose the state once the job is finished.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#39Good Job does the same for Rails
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#40I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project.
Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow.
We all know Celery is a buggy pain in the butt, but rolling your own job queue likely ends up with you just writing a similary-buggy pain in the butt. We've already done "Celery but simpler", it's stuff like Dramatiq!
If you have backend-specific needs, you won't listen to this advice. But think deeply how important your needs are. Computers are fast, and you can deal with a lot of events with most systems.
Meanwhile if you use a backend-generic system... well you could write a backend using PgQueuer!