Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A first look at Django's new background tasks

roam.be

Post reply on HN
Print view View original

21–30 of 49 posts

Re: A first look at Django's new background tasks

#21

Post by pmontra »

elliot07 wrote: post #18

Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?

A customer of mine has two projects. One running on their own hardware, Django + Celery. The other one running on AWS EC2, Django alone.

In the first one we use Celery to run some jobs that may last from a few seconds to some minutes. In the other one we create a new VM and make it run the job and we make it self destroy on job termination. The communication is over a shared database and SQS queues.

We have periodic problems with celery: workers losing connection with RabbitMQ, Celery itself getting stuck, gevent issues maybe caused by C libraries but we can't be sure (we use prefork for some workers but not for everything)

We had no problems with EC2 VMs. By the way, we use VirtualBox to simulate EC2 locally: a Python class encapsulates the API to start the VMs and does it with boto3 in production and with VBoxManage in development.

What I don't understand is: it's always Linux, amd64, RabbitMQ but my other customer using Rails and Sidekiq has no problems and they run many more jobs. There is something in the concurrency stack inside Celery that is too fragile.

Re: A first look at Django's new background tasks

#22

Post by adamcharnock »

Something about queuing systems that often gets me is that they can start to seem like the wrong abstraction as soon as one has tasks that enqueue additional tasks. Particularly when features start growing, and double particularly when modelling business processes.

This is because the code enqueuing the task needs to be aware of what happens next, which breaks separation of concerns. Why should the user sign-up code have to know that a report generation job now needs queuing?

Really what starts to make more sense to me is to fire off events. Code can say, "this thing just happened", and let other code decide if it wants to listen. When then makes it an event stream rather than a queue, with consumer groups at al.

I made the (now unmaintained) project https://lightbus.org around this, and it did work really well for our use case. Hopefully someone has now created something better.

So I'd say this: before grabbing for a task queue, take a moment to think about what you're actually modelling. But be careful of the event streaming rabbit-hole!

Re: A first look at Django's new background tasks

#23

Post by globular-toast »

adamcharnock wrote: post #22

Something about queuing systems that often gets me is that they can start to seem like the wrong abstraction as soon as one has tasks that enqueue additional tasks. Particularly when features start growing, and double particularly when modelling business processes. This is because the code enqueuing the task needs to be aware of what happens next, which breaks separation of concerns. Why should the user sign-up code…

They're not mutually exclusive. Nothing about "event driven" means async. I have an event driven modular monolith and all events are handled synchronously. It's up to the receiver to queue a task if it needs to, so context boundaries are not crossed.

Re: A first look at Django's new background tasks

#26

Post by Grollicus »

That's intersting to see.

Does the api support progress reporting? ("30 % done")

Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that.

Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks.

Re: A first look at Django's new background tasks

#27

Post by ducdetronquito »

Grollicus wrote: post #26

That's intersting to see. Does the api support progress reporting? ("30 % done") Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that. Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks…

To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? Is it even possible to do it accurately ?

Though, I imagine you could have strategies to give an approximation of it, for example like keeping track of the past execution time of a given type of task in order to infer the progress of a currently running task of the same type.

Re: A first look at Django's new background tasks

#28

Post by ducdetronquito »

JakaJancar wrote: post #13

Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great: https://procrastinate.readthedocs.io/en/stable/index.html Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested. IIRC think the codebase is like one-tenth that of Celery.

I also warmly recommend procrastinate !

We moved all our celery tasks to procrastinate at work for all our django backends since almost two years now and it has been great.

Having tasks deferred in the same transaction as the business logic stuff is something that helped us a lot to improve consistency and debugability. Moreover, it's so nice to able to inspect what's going on by just querying our database or just looking at the django admin.

For those wondering, procrastinate has no built-in alternative to django-celery-beat, but you can easily build your own in a day: no need for an extra dependency for this :)

Re: A first look at Django's new background tasks

#30

Post by pbalau »

ducdetronquito wrote: post #27
Grollicus wrote: post #26

That's intersting to see. Does the api support progress reporting? ("30 % done") Of course one could build this manually when you're building the worker implementation but I'd love to have it reflected in the api somewhere. Celery also seems to be missing api for that. Anyone sees a reason why that's missing? I don't think it complicates the api much and seems such an obvious thing for longer-running background tasks…

To implement progress reporting, it means you are able to know the time a task would take to run upfront, no? Is it even possible to do it accurately ? Though, I imagine you could have strategies to give an approximation of it, for example like keeping track of the past execution time of a given type of task in order to infer the progress of a currently running task of the same type.

> To implement progress reporting, it means you are able to know the time a task would take to run upfront, no?

No. You just need to know the total number of steps and what step are you currently on.

Post reply on HN