Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Procrastinate: PostgreSQL-Based Task Queue for Python

procrastinate.readthedocs.io

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#31

Post by ungawatkt »

41b696ef1113 wrote: post #10
shroompasta wrote: post #3

Earlier quoted context omitted.

Why this and not celery? And why Postgres and not rabbitmq or redis?

While I respect Celery, it is complicated (some amount is intrinsic to the problem space). I used to use it, but it felt heavyweight and debugging errors was so quite challenging. Have since switched to Dramatiq - which may not be web scale, but works fine for an internal service. I love the idea of re-using infrastructure "for free".

+1 for dramatiq over celery, dramatiq was much easier to understand and extend, even if it's not as "complete". It handled most anything I threw at it for an internal task runner system with some custom scheduling logic. I'd give it a flask out of 10 compared to celery being a Django.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#32

Post by mmcgaha »

aidos wrote: post #18
mst wrote: post #11

My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.

EDIT: I was understandably being downvoted for saying that the code could be more readable. On reflection, the sql is good, and it's the Perl-ness that I had a bit of a reaction to. I'll leave my questions here in case people would like some food for thought anyway. - what's with the mixture of styles (q{sql}, sql with '?', sql with '$1', sql with '\$1' etc)? - what's the use case for a right join, when left joining…

I agree with you on the right/left join issue. I don't think I have ever used a right join. But then again I don't use greater-than signs either so maybe it is me.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#33

Post by sischoel »

Genuine question, not a comment on OPs implementation: What is the reason that people so often use some external service for creating task queues (often backed by some kind of db), instead of just implementing it themselves in whatever language they are using for the rest of their application?

Is it: - They want persistence in case the system goes down, or the task queue service is restarted.

- They want to replicate their task queue service and want to be sure that the data is properly synchronized.

- The amount of tasks/time if simply too much for some standard library queues

- They think is is easier to add another dependency than implement it themselves

- something else

?

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#34

Post by tibanne »

sischoel wrote: post #33

Genuine question, not a comment on OPs implementation: What is the reason that people so often use some external service for creating task queues (often backed by some kind of db), instead of just implementing it themselves in whatever language they are using for the rest of their application? Is it: - They want persistence in case the system goes down, or the task queue service is restarted. - They want to replicate…

I can't say, I implemented my own queuing system in Python. It seemed simpler.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#35

Post by aidos »

gurjeet wrote: post #20
aidos wrote: post #18

Earlier quoted context omitted.

EDIT: I was understandably being downvoted for saying that the code could be more readable. On reflection, the sql is good, and it's the Perl-ness that I had a bit of a reaction to. I'll leave my questions here in case people would like some food for thought anyway. - what's with the mixture of styles (q{sql}, sql with '?', sql with '$1', sql with '\$1' etc)? - what's the use case for a right join, when left joining…

Sorry to be that guy, but your comment is pointless, so I had to downvote it. While the beauty is in the eye of beholder, and you may be correct, or have high standards, praise coming from the likes of @mst should be taken seriously. People may have seen code that can stand in for sh?t, so it makes them appreciate simple niceties like good formatting, good and concise logic, etc. BTW, I have to agree with @mst on thi…

I think your comment here is reasonable and you were unfairly downvoted by others missing the deleted context in my original comment.

I’ve upvoted you partly on those grounds, and partly to get you back to 1337 :)

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#36

Post by gurjeet »

aidos wrote: post #35
gurjeet wrote: post #20

Earlier quoted context omitted.

Sorry to be that guy, but your comment is pointless, so I had to downvote it. While the beauty is in the eye of beholder, and you may be correct, or have high standards, praise coming from the likes of @mst should be taken seriously. People may have seen code that can stand in for sh?t, so it makes them appreciate simple niceties like good formatting, good and concise logic, etc. BTW, I have to agree with @mst on thi…

I think your comment here is reasonable and you were unfairly downvoted by others missing the deleted context in my original comment. I’ve upvoted you partly on those grounds, and partly to get you back to 1337 :)

Hah. I was bummed when I lost my leet score, especially because it was lost towards the wrong direction.

Thank you!

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#37

Post by nhumrich »

sischoel wrote: post #33

Genuine question, not a comment on OPs implementation: What is the reason that people so often use some external service for creating task queues (often backed by some kind of db), instead of just implementing it themselves in whatever language they are using for the rest of their application? Is it: - They want persistence in case the system goes down, or the task queue service is restarted. - They want to replicate…

Distributed queuing, with horizontal scaling workers and retries, without ever loosing a message/job, is surprisingly difficult to get right. It's simply not worth your time in a lot of cases. Similarly, why people don't just write their own web framework.

It sounds like your question is focused on "why not just use an in memory queue". Two reasons: 1. You loose messages/jobs, and it's not scalable 2. If the task is CPU intensive, it will lock up or slow down your web server.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#38

Post by tpoacher »

Not making a comparison, as I haven't used Procrastinate, but the author doesn't mention luigi as an alternative/competitor (originally developed at spotify, if I'm not mistaken). I thought luigi was a relatively big player in the python 'task management' scene; I have used luigi and it was incredibly powerful, and fairly intuitive to use ... was a bit surprised not to see it mentioned.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#39

Post by mst »

aidos wrote: post #18
mst wrote: post #11

My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.

EDIT: I was understandably being downvoted for saying that the code could be more readable. On reflection, the sql is good, and it's the Perl-ness that I had a bit of a reaction to. I'll leave my questions here in case people would like some food for thought anyway. - what's with the mixture of styles (q{sql}, sql with '?', sql with '$1', sql with '\$1' etc)? - what's the use case for a right join, when left joining…

I see where you're coming from finding the perlish string formatting weird, I guess I barely even notice that part and was assuming incorrectly it would be everything else about the perl that would bother people.

So I guess that while you perhaps did have suboptimal judgement in your initial reply you very much weren't alone and I should remember to be more careful about that going forwards.

Re: Procrastinate: PostgreSQL-Based Task Queue for Python

#40

Post by mst »

setr wrote: post #23
mst wrote: post #11

My current favourite task queue is http://p3rl.org/Minion and I mention this not in a spirit of competition but because the pg queries to be found in the source code of https://metacpan.org/dist/Minion/source/lib/Minion/Backend/P... are IMO truly beautiful and worthy of the "great artists steal" treatment.

I’m reading through that file and I’m not finding anything particularly interesting — the queries all look… reasonable — though I’m also struggling to imagine what I would consider a beautiful query Anything specific worth calling out?

I suspect as much as anything else this is a case of "any piece of code I can read through and not have a viscerally negative reaction to part of it that makes me want to rewrite it to within an inch of its life counts as beautiful to me".

Certainly, very little of my own code passes that test after I've not looked at it for a week.

Post reply on HN