Procrastinate: PostgreSQL-Based Task Queue for Python
procrastinate.readthedocs.io
Procrastinate: PostgreSQL-Based Task Queue for Python
1–10 of 40 posts
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#2I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1].
I thought it seemed like a project deserving a bit more attention.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#3I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery?
And why Postgres and not rabbitmq or redis?
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#4This looks nice, I'll give it a try.
It's worth mentionning Huey too: https://huey.readthedocs.io/en/latest/ I've been using it for a year on top of sqlite.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#5I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery? And why Postgres and not rabbitmq or redis?
Re: why Postgres, two common answers:
1. For anyone whose stack is currently a simple three-tier architecture that currently has Postgres and nothing else in the DB tier, adding anything else would incur 100+% operational complexification (e.g. now you have to figure out how to do backups for two stateful components.) Far more than 100%, even, if they get the Postgres DB "for free" as part of a virtual LAMP-stack appliance, or built into the base offering of some PaaS service, and currently don't need to do any ops work as the framework/platform handles their DB's care and feeding for them. (Ideally, such setups would do the same with a built-in MQ as well — but sadly, that's much rarer.)
2. You can be clever with a job queue that’s in your DB, by making “taking the job” and “doing the queries that comprise the job” part of the same atomic transaction, such that a ROLLBACK due to a constraint validation error in "the queries that comprise the job" will also implicitly “put back” the job immediately (even if the worker crashed in response to seeing the error.)
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#6How does this compare to pq?
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#7I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery? And why Postgres and not rabbitmq or redis?
Not parent but in my experience celery has a number of long standing bugs / poor defaults, like prefetching tasks so they can get stuck behind long running tasks. Next project id probably try a simplistic project that’s easier to understand like arq / rq.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#8The documentation is really good. I particularly like this page, which goes into detail about why they built it and how it is designed: https://procrastinate.readthedocs.io/en/stable/discussions.h...
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#9I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery? And why Postgres and not rabbitmq or redis?
They address that on this page: https://procrastinate.readthedocs.io/en/stable/discussions.h...
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#10I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery? And why Postgres and not rabbitmq or redis?
While I respect Celery, it is complicated (some amount is intrinsic to the problem space). I used to use it, but it felt heavyweight and debugging errors was so quite challenging. Have since switched to Dramatiq - which may not be web scale, but works fine for an internal service.
I love the idea of re-using infrastructure "for free".