Django this is about 10 years too late. It's frustrating because we use all manner of hacks to work around this being part of the builtin story.
A first look at Django's new background tasks
11–20 of 49 posts
Re: A first look at Django's new background tasks
#12How is the typing support? We just had downtime because a change to a celery task didn't trigger mypy to complain for all call sites until runtime. Too many python decorators aren't written with pretty weak typing support.
Static analysis will never be fully robust in Python. As a simple example, you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically, or even know what the call path of the functions is, without actually running the code in trace/profiler mode. You probably want something like pydantic’s @validate_call decorator.
> you can define a function that only exists at runtime, so even in principle it wouldn’t be possible to type check that statically
Can you say more, maybe with with an example, about a function which can't be typed? Are you talking about generating bytecode at runtime, defining functions with lambda expressions, or something else?
Re: A first look at Django's new background tasks
#13Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great:
https://procrastinate.readthedocs.io/en/stable/index.html
Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested.
IIRC think the codebase is like one-tenth that of Celery.
Re: A first look at Django's new background tasks
#14Assuming you're fine with keeping the queue in postgres, I've used Procrastinate and it's great: https://procrastinate.readthedocs.io/en/stable/index.html Core is not Django-specific, but it has an optional integration. Sync and async, retries/cancellation/etc., very extensible, and IMO super clean architecture and well tested. IIRC think the codebase is like one-tenth that of Celery.
If you like Procastinate, you might like my Chancy, which is also built on postgres but with a goal of the most common bells and whistles being included.
Rate limiting, global uniqueness, timeouts, memory limits, mix asyncio/processes/threads/sub-interpreters in the same worker, workflows, cron jobs, dashboard, metrics, django integrations, repriotization, triggers, pruning, Windows support, queue tagging (ex: run this queue on all machines running windows with a GPU, run this one on workers with py3.14 and this one on workers with py3.11) etc etc...
https://tkte.ch/chancy/ & https://github.com/tktech/chancy
The pending v0.26 includes stabilizing of the HTTP API, dashboard improvements, workflow performance improvements for workflows with thousands of steps and django-tasks integration.
Re: A first look at Django's new background tasks
#15Really cool to see a batteries‑included option in Django for background jobs.
For folks who’ve used Celery/Procrastinate/Chancy: how does retry/ACK behavior feel in real projects? Any rough edges?
What about observability — dashboards, tracing, metrics — good enough out of the box, or did you bolt on extra stuff?
Also, any gotchas with type hints or decorator-style tasks when refactoring? I’ve seen those bite before.
And lastly, does swapping backends for tests actually feel seamless, or is that more of a “works in the demo” thing?
Re: A first look at Django's new background tasks
#16Really cool to see a batteries‑included option in Django for background jobs. For folks who’ve used Celery/Procrastinate/Chancy: how does retry/ACK behavior feel in real projects? Any rough edges? What about observability — dashboards, tracing, metrics — good enough out of the box, or did you bolt on extra stuff? Also, any gotchas with type hints or decorator-style tasks when refactoring? I’ve seen those bite before.…
(I'm biased, I'm the author of Chancy)
One of the major complaints with Celery is observability. Databased-backed options like Procastinate and Chancy will never reach the potential peak throughput of Celery+RabbitMQ, but they're still sufficient to run millions upon millions of tasks per day even on a $14/month VPS. The tradeoff to this is excellent insight into what's going on - all state lives in the database, you can just query it. Both Procastinate and Chancy come with Django integrations, so you can even query it with the ORM.
For Chancy in particular, retries are a (very trivial) plugin (that's enabled by default) - https://github.com/TkTech/chancy/blob/main/chancy/plugins/re.... You can swap it out and add whatever complex retry strategies you'd like.
Chancy also comes with a "good enough" metrics plugin and a dashboard. Not suitable for an incredibly busy instance with tens of thousands of distinct types of jobs, but good enough for most projects. You can see the new UI and some example screenshots in the upcoming 0.26 release - https://github.com/TkTech/chancy/pull/58 (and that dashboard is for a production app running ~600k jobs a day on what's practically a toaster). The dashboard can be run standalone locally and pointing to any database as-needed, run inside a worker process, or embedded inside any existing asgi app.
Re: A first look at Django's new background tasks
#17I’ve been using the django-tasks library in production for about a year. The database backend and simple interface have been great. It definitely isn’t intended to replace all of celery, but for a simple task queue that doesn’t require additional infrastructure it works quite well.
This one? https://github.com/RealOrangeOne/django-tasks
That and the rq backend sound promising to me.
Re: A first look at Django's new background tasks
#18Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well.
Anyone here done the migration off of celery to another thing? Any wisdom?
Re: A first look at Django's new background tasks
#19Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?
We switched from Celery to Temporal. Temporal is such a great piece of distributed system.
Re: A first look at Django's new background tasks
#20Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?
Migrated Celery to Argo Workflows. No wisdom as it was straightforward. You lose a lot startup speed though, so it's not a drop-in replacement and is only a good choice for long-running workflows. Celery was easier than Argo Workflows. Celery is really easy to get started with. I like Airflow the best, but it's closer to Argo Workflows in terms of more long-lived workflows. I hope to try Hatchet soon. I've read Temporal is even harder to manage.