71–80 of 140 posts
Cool, congrats on releasing. Have you seen graphile worker? Wondering how this compares or if you're building for different use-cases.
Good Job does the same for Rails https://github.com/bensheldon/good_job
There’s also the new built in SolidQueue. https://github.com/rails/solid_queue/
Earlier quoted context omitted.
> It's simple and atomic and almost certainly incorrect, gotta read at least https://www.pgcon.org/2016/schedule/attachments/414_queues-p... which discusses FOR UPDATE SKIP LOCKED
that's a nice read, but does it also apply to MySQL (InnoDB)?
Earlier quoted context omitted.
that's a nice read, but does it also apply to MySQL (InnoDB)?
I've done this with mysql. Never do one at a time if you have jobs per minute over 30. It won't scale. Instead have the job dispatcher reserve 100 at a time and then fire that off to a subprocess which will subsequently fire off a process for each job. A three layer approach makes it much easier to build out multiserver. Or if you don't want the headache just use SQS which is pretty much free under 1 million jobs.
However if I am about to use DB as a job queue for budget reasons, I'd make sure the job doesn't get too complicated.
Earlier quoted context omitted.
I've done this with mysql. Never do one at a time if you have jobs per minute over 30. It won't scale. Instead have the job dispatcher reserve 100 at a time and then fire that off to a subprocess which will subsequently fire off a process for each job. A three layer approach makes it much easier to build out multiserver. Or if you don't want the headache just use SQS which is pretty much free under 1 million jobs.
Yeah it's very basic and limited. However if I am about to use DB as a job queue for budget reasons, I'd make sure the job doesn't get too complicated.
I was able to get it up to 3500 jobs an hour and likely could have gone far past that but the load on the MySQL server was not reasonable
It also supports many other backends including AMQP, Beanstalkd, Redis and various cloud services.
This component, called Messenger, can be installed as a standalone library in any PHP project.
(Disclaimer: I’m the author of the PostgreSQL transport for Symfony Messenger).
Does PgQueuer address any of them?
This is very interesting tool.
You might also want to look at River ( https://github.com/riverqueue/river ) for inspiration as they support scheduled jobs, etc. From an end-user perspective, they also have a UI which is nice to have for debugging.
Although seems like OP references a Python library rather than standalone server, so would probably be useful to Python devs.
There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and use polling). It also supports many features (and more are planned), like sync and async jobs (it uses asyncio under the hood), periodic tasks, retries, task locks, priorities, job cancellation/aborting, Django integration (optional). DISCLAIME…