41–50 of 140 posts
You might also want to look at River ( https://github.com/riverqueue/river ) for inspiration as they support scheduled jobs, etc. From an end-user perspective, they also have a UI which is nice to have for debugging.
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
This would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.
Cool, congrats on releasing. Have you seen graphile worker? Wondering how this compares or if you're building for different use-cases.
Any suggestions for something like this for dotnet?
I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…
Define "operationally simple", most if not all of them need persistent anyway, on top of the queue itself. This eliminates the queue and uses a persistent you likely already have.
There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and use polling). It also supports many features (and more are planned), like sync and async jobs (it uses asyncio under the hood), periodic tasks, retries, task locks, priorities, job cancellation/aborting, Django integration (optional). DISCLAIME…
I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…
> those systems offer good affordances for testing and running locally in an operationally simple way Define "operationally simple", most if not all of them need persistent anyway, on top of the queue itself. This eliminates the queue and uses a persistent you likely already have.
How many times have you shipped some background task change, only to realize half your test suite doesn't do anything with background tasks, and you're not testing your business logic to the logical conclusion? Eager task execution catches bugs earlier on, and is close enough to the reality for things that matter, while removing the need for, say, multi-process cordination in most tests.
And you can still test things the "real way" if you need to!
And to your other point: you can use Dramatiq with Postgres, for example[0]. I've written custom backends that just use pg for these libs, it's usually straightforward because the broker classes tend to abstract the gnarly things.
I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…
I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)
Earlier quoted context omitted.
At low-medium scale, this will be fine. Even at higher scale, so long as you monitor autovacuum performance on the queue table. At some point it may become practical to bring a dedicated queue system into the stack, sure, but this can massively simplify things when you don’t need or want the additional complexity.
Aside from that, the main advantage of this is transactions. I can do: begin; insert_row(); schedule_job_for_elasticsearch(); commit; And it's guaranteed that both the row and job for Elasticsearch update are inserted. If you use a dedicated queue system them this becomes a lot more tricky: begin; insert_row(); schedule_job_for_elasticsearch(); commit; // Can fail, and then we have a ES job but no SQL row. begin; ins…
And as far as I can tell, this is only a perk when your two actions are mutate the collocated database and do X. For all other situations this seems like a downgrade.
Earlier quoted context omitted.
I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)
Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?
The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement an immediate notification mechanism if there was a protocol that supported it.