Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A first look at Django's new background tasks

roam.be

Post reply on HN
Print view View original

41–49 of 49 posts

Re: A first look at Django's new background tasks

#41

Post by icedchai »

elliot07 wrote: post #18

Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?

I remember working at a startup in the early 2010's that used celery for all of its background job infrastructure. There were several million tasks run daily across dozens of servers. Celery would regularly hang. Queues would pile up. We had some crazy scripts that would restart celery, detect and kill hanging processes, etc. Fun times.

Re: A first look at Django's new background tasks

#42

Post by amanzi »

Grollicus wrote: post #26

That's intersting to see. Does the api support progress reporting? ("30 % done") Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that. Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks…

I had previously asked about this. My understanding is that there is a desire to add functionality to allow this, but in this first release the task system is intentionally very lightweight.

Re: A first look at Django's new background tasks

#43

Post by Grollicus »

amanzi wrote: post #42
Grollicus wrote: post #26

That's intersting to see. Does the api support progress reporting? ("30 % done") Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that. Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks…

I had previously asked about this. My understanding is that there is a desire to add functionality to allow this, but in this first release the task system is intentionally very lightweight.

Do you have any source one could follow? mail thread, bug tracker issue, ..?

Re: A first look at Django's new background tasks

#44

Post by ducdetronquito »

pbalau wrote: post #30
ducdetronquito wrote: post #27

Earlier quoted context omitted.

To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? Is it even possible to do it accurately ? Though, I imagine you could have strategies to give an approximation of it, for example like keeping track of the past execution time of a given type of task in order to infer the progress of a currently running task of the same type.

> To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? No. You just need to know the total number of steps and what step are you currently on.

You're right, but I also don't find a way a task queue library could know that upfront either to implement progress reporting.

Does anyone know a task queue library that implement it ? I would be curious to look at it !

Re: A first look at Django's new background tasks

#45

Post by amanzi »

Grollicus wrote: post #43
amanzi wrote: post #42

Earlier quoted context omitted.

I had previously asked about this. My understanding is that there is a desire to add functionality to allow this, but in this first release the task system is intentionally very lightweight.

Do you have any source one could follow? mail thread, bug tracker issue, ..?

This is the repo where the development of django-tasks is happening: https://github.com/realOrangeOne/django-tasks

Re: A first look at Django's new background tasks

#46

Post by amanzi »

ducdetronquito wrote: post #44
pbalau wrote: post #30

Earlier quoted context omitted.

> To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? No. You just need to know the total number of steps and what step are you currently on.

You're right, but I also don't find a way a task queue library could know that upfront either to implement progress reporting. Does anyone know a task queue library that implement it ? I would be curious to look at it !

The way it's typically done is that the worker process reports back its progress to the job metadata on the queue, and the web worker polls the job metadata to read the progress. I've implemented this for progress bars many times on Django with django-rq.

I first learned about it from Miguel Grinberg's Flask tutorial: https://blog.miguelgrinberg.com/post/the-flask-mega-tutorial... But the same concept applies to Django.

Re: A first look at Django's new background tasks

#47

Post by JakaJancar »

ducdetronquito wrote: post #28
JakaJancar wrote: post #13

Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great: https://procrastinate.readthedocs.io/en/stable/index.html Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested. IIRC think the codebase is like one-tenth that of Celery.

I also warmly recommend procrastinate ! We moved all our celery tasks to procrastinate at work for all our django backends since almost two years now and it has been great. Having tasks deferred in the same transaction as the business logic stuff is something that helped us a lot to improve consistency and debugability. Moreover, it's so nice to able to inspect what's going on by just querying our database or just lo…

Procrastinate does have scheduled tasks but maybe I’m missing some other part of beat you mean (never used celery)
Post reply on HN