Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A first look at Django's new background tasks

roam.be

Post reply on HN
Print view View original

31–40 of 49 posts

Re: A first look at Django's new background tasks

#31

Post by jimbohn »

elliot07 wrote: post #18

Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?

Can share the sentiment, had to work with celery years ago, and the maintenance/footguns exceeded the expectations. The codebase and docs are also a bit messy, it's a huge project used and contributed by many so it's understandable I guess. Anyway, Argo if you are in K8S, something else if you aren't. And if you are a startup and need speed, just go with something like procrastinate.

Re: A first look at Django's new background tasks

#32

Post by leonkenneth »

Very excited to see these kind of improvements. Coming from Rails to Django I've been a bit disappointed in the amount of things provided out of the box for a SaaS app (the main exception being of course the admin). Even in the ecosystem, I find packages to be a bit more messy and less production-ready than in Rails. Granted I'm fairly new to it, and the design philosophies are different

Re: A first look at Django's new background tasks

#33

Post by stuaxo »

pmontra wrote: post #21
elliot07 wrote: post #18

Celery is such garbage to run/maintain at any sort of scale. Very excited for this. Rq/temporal also seem to solve this well. Anyone here done the migration off of celery to another thing? Any wisdom?

A customer of mine has two projects. One running on their own hardware, Django + Celery. The other one running on AWS EC2, Django alone. In the first one we use Celery to run some jobs that may last from a few seconds to some minutes. In the other one we create a new VM and make it run the job and we make it self destroy on job termination. The communication is over a shared database and SQS queues. We have periodic…

Celery with the Redis backend seemed pretty solid.

RabbitMQ and friends are just a pain to use.

Re: A first look at Django's new background tasks

#34

Post by stuaxo »

TkTech wrote: post #16
frankhsu wrote: post #15

Really cool to see a batteries‑included option in Django for background jobs. For folks who’ve used Celery/Procrastinate/Chancy: how does retry/ACK behavior feel in real projects? Any rough edges? What about observability — dashboards, tracing, metrics — good enough out of the box, or did you bolt on extra stuff? Also, any gotchas with type hints or decorator-style tasks when refactoring? I’ve seen those bite before.…

(I'm biased, I'm the author of Chancy) One of the major complaints with Celery is observability. Databased-backed options like Procastinate and Chancy will never reach the potential peak throughput of Celery+RabbitMQ, but they're still sufficient to run millions upon millions of tasks per day even on a $14/month VPS. The tradeoff to this is excellent insight into what's going on - all state lives in the database, you…

Celery Flower has some rough corners but generally I can manage celery tasks there and see where things are failing or backed up.

Re: A first look at Django's new background tasks

#35

Post by stuaxo »

the__alchemist wrote: post #6

This is great! The prev recommendation was usually a lib called celery that I wasn't able to get working. I don't remember the details, but it had high friction points or compatibility barriers I wasn't able to overcome. This integration fits Django's batteries included approach. I've been handling this, so far, with separate standalone scripts that hook into Django's models and ORM. You have to use certain incantati…

Celery generally works best with the redis backend.

The main thing when it comes to models is that you pass the ids not the model instances themselves when passing to celery jobs.

This is because the celery worker could be running somewhere else entirely.

Re: A first look at Django's new background tasks

#38

Post by halfcat »

adamcharnock wrote: post #22

Something about queuing systems that often gets me is that they can start to seem like the wrong abstraction as soon as one has tasks that enqueue additional tasks. Particularly when features start growing, and double particularly when modelling business processes. This is because the code enqueuing the task needs to be aware of what happens next, which breaks separation of concerns. Why should the user sign-up code…

It sounds like the need you’re describing is not event-driven, but more like orchestration where the orchestrator is aware of dependencies between tasks and runs them in the right order (it builds a DAG, a directed acyclic graph). Tools like Airflow do this.

Re: A first look at Django's new background tasks

#39

Post by halfcat »

benatkin wrote: post #17
vforgione wrote: post #8

I’ve been using the django-tasks library in production for about a year. The database backend and simple interface have been great. It definitely isn’t intended to replace all of celery, but for a simple task queue that doesn’t require additional infrastructure it works quite well.

This one? https://github.com/RealOrangeOne/django-tasks That and the rq backend sound promising to me.

Yes, the RealOrangeOne repo was the working repo before it got merged into Django. It’s the same thing the article is talking about.

Re: A first look at Django's new background tasks

#40

Post by benatkin »

halfcat wrote: post #39
benatkin wrote: post #17

Earlier quoted context omitted.

This one? https://github.com/RealOrangeOne/django-tasks That and the rq backend sound promising to me.

Yes, the RealOrangeOne repo was the working repo before it got merged into Django. It’s the same thing the article is talking about.

Has it been merged into the GitHub repo? I wasn't aware of that and I don't see it here: https://github.com/django/django/tree/main/django/tasks/back...
Post reply on HN