I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
man... couldnt agree more.
Celery in production: Three more years of fixing bugs
11–20 of 63 posts
Re: Celery in production: Three more years of fixing bugs
#12I have been using Celery a lot for several projects and haven't really run into any significant issues. I like it as part of my tech stack.
Sorry to hear others don't feel the same way. I find it easy to use and maintain.
Re: Celery in production: Three more years of fixing bugs
#13RQ [0] (Python-based with Redis backend) is another option; works great.
[0] https://python-rq.org/
Re: Celery in production: Three more years of fixing bugs
#14> Supervisor should've killed celery's forked workers once the parent dies. Either it is not reliable or we haven't been able to make this happen in at least a few attempts now. Use systemd instead of supervisord for misbehaving forking daemons like this. If you set "KillMode=mixed" you get a nice compromise between "signal the parent and give things a configurable grace period to shut themselves down" and "zero risk…
I find supervisor's event listener functionality useful for implementing watchdog processes. I have the supervised process write an event whenever it successfully completes a task. Meanwhile, a separate event listener process has a watchdog timer which asks supervisor to restart the supervised process if it hasn't checked in recently enough.
Does systemd have anything like supervisor events?
Re: Celery in production: Three more years of fixing bugs
#15I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.
Re: Celery in production: Three more years of fixing bugs
#16I love Huey [1]. I switched everything over from Celery to it. It’s so simple it’s a godsend, even supports SQLite queue [1] https://github.com/coleifer/huey
Thoughts on Huey vs RQ?
Re: Celery in production: Three more years of fixing bugs
#17I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.
Reddit :)
Re: Celery in production: Three more years of fixing bugs
#18I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
What would you recommend instead?
I'm also in the 'anything but Celery' camp:
https://dramatiq.io/motivation.html
If you're backing onto something like SQS or Rabbit (Redis is a bit trickier), it's not the worst thing to hand-roll either. Then you understand exactly what trade-offs you're making.
Re: Celery in production: Three more years of fixing bugs
#19I wrote an extremely performant and simple async worker framework called SAQ because I couldn't find any that fit my use case.
Re: Celery in production: Three more years of fixing bugs
#20> Supervisor should've killed celery's forked workers once the parent dies. Either it is not reliable or we haven't been able to make this happen in at least a few attempts now. Use systemd instead of supervisord for misbehaving forking daemons like this. If you set "KillMode=mixed" you get a nice compromise between "signal the parent and give things a configurable grace period to shut themselves down" and "zero risk…
I find supervisor's event listener functionality useful for implementing watchdog processes. I have the supervised process write an event whenever it successfully completes a task. Meanwhile, a separate event listener process has a watchdog timer which asks supervisor to restart the supervised process if it hasn't checked in recently enough. Does systemd have anything like supervisor events? http://supervisord.org/ev…
Yes, sd_notify(0, "WATCHDOG=1") should get you started.