The most simple job queue in MySQL: update job_table set key=value where ... limit 1 It's simple and atomic. Unfortunately PG doesn't allow `update ... limit` syntax
[deleted]
Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
111–120 of 140 posts
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#112Good Job does the same for Rails https://github.com/bensheldon/good_job
[deleted]
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#113There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and use polling). It also supports many features (and more are planned), like sync and async jobs (it uses asyncio under the hood), periodic tasks, retries, task locks, priorities, job cancellation/aborting, Django integration (optional). DISCLAIME…
[deleted]
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#114I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq .
What I like about https://github.com/tembo-io/pgmq is that it can be used with any programming language and does not require any background workers or running a binary in addition to postgres.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#115I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…
Some names
- Celery (massive and heavy)
- Dramatiq
- APScheduler
- Huey
Today, Redis queues, unless stricly a single process, seem to be most pain free for small scale use.
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#116I really like the emergence of simple queuing tools for robust database management systems. Keep things simple and remove infrastructure complexity. Definitely a +1 from me!
For handling straightforward asynchronous tasks like sending opt-in emails, we've developed a similar library at All Quiet for C# and MongoDB: https://allquiet.app/open-source/mongo-queueing
In this context:
LISTEN/NOTIFY in PostgreSQL is comparable to MongoDB's change streams.
SELECT FOR UPDATE SKIP LOCKED in PostgreSQL can be likened to MongoDB's atomic read/update operations.Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#117I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
A common schema is one nice thing, but imho the win of these db backed queues is being able to do things, including enqueue background jobs in a single transaction. e.g. create user, enqueue welcome email - both get done, or not - with redid-based, this is ... not usually a thing; if you fail to do one, it's left half done, leading to more code etc
p.s. I maintain a ruby equivalent called QueueClassic
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#118I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…
In my experience, it's easy to test locally with PG: we have unit tests which re-create DB for each test... It works.
Also DB transactions are absolutely the best way to provide ACID guarantee
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#119Most of the small python alternatives I've seen use Redis as backend:
- https://github.com/rq/rq - https://github.com/coleifer/huey - https://github.com/resque/resque
Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue
#120Most of the small python alternatives I've seen use Redis as backend: - https://github.com/rq/rq - https://github.com/coleifer/huey - https://github.com/resque/resque
[deleted]