My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.
Procrastinate: PostgreSQL-Based Task Queue for Python
11–20 of 40 posts
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#12The documentation is really good. I particularly like this page, which goes into detail about why they built it and how it is designed: https://procrastinate.readthedocs.io/en/stable/discussions.h...
That page is beautiful and the architecture is clearly quite similar to perl's Minion that I mentioned in a top level comment and frankly the more implementations of such a concept in different languages the better and <3
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#13I’ve been looking for a tool that could replace Celery for my smaller projects and use postgresql as its queuing system. I was pointed this way in the recent thread about message queuing natively in postgresql [1]. I thought it seemed like a project deserving a bit more attention. https://news.ycombinator.com/item?id=30119285
Why this and not celery? And why Postgres and not rabbitmq or redis?
> why Postgres and not rabbitmq or redis?
I chose postrges for our mq stack over rabbitmq and redis. Since our team is small, we are trying to be efficient in our use of manpower. our architecture is as dirt simple as it gets. (loadbalancer -> api-server-nodes -> aws aurora).
adding another external dependency just adds one more thing that can potentially go wrong. Thats one more thing we need to watch and maintain accross staging and production envrionemnts and one more thingto get running on developer machines.
There may come a day where we switch over to rabbitmq or kafka but postgres has proven to be fast enough. when writes are too much of a load, we can shard the writes to its own dedicated machine and buy more time. When our traffic is sufficiently high that even that isn't enough, we'll have enough revenue to pay someone to configure kafka/rabbitmq and deal with the configuration and maintenance fulltime.
Until then, if you want to survive as a startup, KEEP IT SIMPLE. Any complexity you adopt needs to justify itself by providing a competitive advantage.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#14Quite timely to see this discussion here. I got to a point on a home project last night where I need to implement a scheduler and was thinking 'man I dont want to setup Celery for this', that can wait for tomorrow.
In the past I have actually written my own very little scheduler that borrows the app database. Really just table that is a job queue and a script on a loop that grabs the job when it appears in the table. But it did have the occasional hiccup.
I can't wait to check this out and some other suggestions in the comments.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#15Postgrastinate.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#16Is there a job/message queue implemented in pure Postgres, so it doesn't depend on Python, Perl, Ruby, ...? So that it can be used from any language that can connect to Postgres?
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#17Earlier quoted context omitted.
Why this and not celery? And why Postgres and not rabbitmq or redis?
Not parent but in my experience celery has a number of long standing bugs / poor defaults, like prefetching tasks so they can get stuck behind long running tasks. Next project id probably try a simplistic project that’s easier to understand like arq / rq.
That prefetching is still an issue? I recall it was caused by rabbitmq though? I switched to rq over celery years ago because it was a deal breaker. We have just long running tasks and it didn’t work for us at all. Rq has been great btw.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#18My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.
EDIT: I was understandably being downvoted for saying that the code could be more readable. On reflection, the sql is good, and it's the Perl-ness that I had a bit of a reaction to.
I'll leave my questions here in case people would like some food for thought anyway.
- what's with the mixture of styles (q{sql}, sql with '?', sql with '$1', sql with '\$1' etc)?
- what's the use case for a right join, when left joining is the norm?
- in the $stats query, inactive_workers isn't actually inactive_workers and instead it's patched based on active workers afterwards. For me, the jumping of inconsistent column selections on each line makes it hard to see that. (If someone just copied the sql into psql to run it, it would be wrong, but not obviously)
- Maybe it's not fair, because I have sqlalchemy to improve readability, but this whole thing where the variables aren't obvious in a blob of sql could definitely be easier to read
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#19Earlier quoted context omitted.
Why this and not celery? And why Postgres and not rabbitmq or redis?
Re: why Postgres, two common answers: 1. For anyone whose stack is currently a simple three-tier architecture that currently has Postgres and nothing else in the DB tier, adding anything else would incur 100+% operational complexification (e.g. now you have to figure out how to do backups for two stateful components.) Far more than 100%, even, if they get the Postgres DB "for free" as part of a virtual LAMP-stack app…
Generally yes, adding new tech can complicate your infrastructure, but specifically redis is one of those rare gems that, “just works”.
If you use Celery, the time saved dropping Celery will more than cover the operational costs of redis.
Really, one of the magic tools of the 21st century.
Re: Procrastinate: PostgreSQL-Based Task Queue for Python
#20My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.
EDIT: I was understandably being downvoted for saying that the code could be more readable. On reflection, the sql is good, and it's the Perl-ness that I had a bit of a reaction to. I'll leave my questions here in case people would like some food for thought anyway. - what's with the mixture of styles (q{sql}, sql with '?', sql with '$1', sql with '\$1' etc)? - what's the use case for a right join, when left joining…
Sorry to be that guy, but your comment is pointless, so I had to downvote it.
While the beauty is in the eye of beholder, and you may be correct, or have high standards, praise coming from the likes of @mst should be taken seriously. People may have seen code that can stand in for sh?t, so it makes them appreciate simple niceties like good formatting, good and concise logic, etc.
BTW, I have to agree with @mst on this one; even though perl is very low on my scale of pleasantness for languages, the SQL in that code is beautifully written.