We switched our python task processing from celery to pubsub (on GCP) backed by redis for message storage, and it's much better. Celery is full of bugs and quirks, nobody ever really knew what it was doing when it would fail. For simple task processing, we find it helpful to have a task queue that you can fully understand. Just a hundred lines of code for the subscriber and publisher logic, which we wrote ourselves. If you can get by without workflow and orchestration, pubsub works well at high volume and it's cheap. Something to consider!
Celery in production: Three more years of fixing bugs
51–60 of 63 posts
Re: Celery in production: Three more years of fixing bugs
#52I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
IMO using python at all for this kinda setup with lots of workers is kinda crazy. As there is no usable threading, each worker need its own process. And as the article says, that's a couple of hundred mb of ram per worker. But in other languages, you could have hundreds of threads doing work with the same resources.
Re: Celery in production: Three more years of fixing bugs
#53I'll take the other side of all this hate. We use celery for quite a bit, and I struggle to sympathize with TFA.
Much of what's written comes back to using old python2 libraries or just complaining that celery doesn't have defaults that better suit you. The very first issue mentioned, about not manually setting the concurrency level...this is basic stuff. It always boggles my mind when people deploy something to prod with the copy/paste "get started" instructions from github. Why do people deploy critical infra layers without spending a lot of time to understand the thing they're depending on?
There are genuinely bugs in celery, and task queues are particularly hard to get right, especially with pluggable backends. But if you RTFD and follow good devops, you will have a perfectly fine time.
Re: Celery in production: Three more years of fixing bugs
#54I can't say I've encountered the same things and I've been using Celery since around 2015.
In one case of using it I am routinely throwing work at it. Its main job is to contact third party APIs and parse up to 600MB of XML responses. It's not high traffic but it's active enough to be running jobs often.
This box has been really stable, often times going 6 months without having to restart Flask or Celery because it's humming along without issues and the service itself is fairly stable so it's not getting new versions rolled out (just security patches as needed). It's not a legacy system either, it's just feature complete.
In the past I've had other sites where Celery was doing around a million jobs a month to massage a small amount of data and write it to a Postgres database, it also had no issues and was super stable.
I mainly use standard Celery features backed by Redis on a low'ish end VPS. I would like to know more details about what folks are doing where they're encountering all of these problems.
Re: Celery in production: Three more years of fixing bugs
#55Can any explain why we need task queue? It is unique to Python because of it lack of real threading?
Threading is local to a machine; task queues are generally a solution to distributing a large number of tasks across many distinct compute nodes by using some remote network service (redis, rmq, etc) to track the progress of jobs.
Re: Celery in production: Three more years of fixing bugs
#56Earlier quoted context omitted.
I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.
Reddit :)
That's embarrassing! I'd like to blame autocorrect but I doubt it...
Re: Celery in production: Three more years of fixing bugs
#57I'll take the other side of all this hate. We use celery for quite a bit, and I struggle to sympathize with TFA. Much of what's written comes back to using old python2 libraries or just complaining that celery doesn't have defaults that better suit you. The very first issue mentioned, about not manually setting the concurrency level...this is basic stuff. It always boggles my mind when people deploy something to prod…
The moment I saw mention of Python 2 I was like "well, yeah". I'm sure that won't solve all, or even many of the ills, but if you're not reasonably up to date with your language and libraries then yeah, you're probably gonna have a harder time than you need to.
Re: Celery in production: Three more years of fixing bugs
#58Earlier quoted context omitted.
I love RQ. Nothing but good experiences using it several times over the past ~6 years. However if you are doin mission critical work in your queue that cannot be replayed Celery is still the best python flavored option IMO
Others above have been suggesting Huey as an alternative. Do you know of any reason that it falls short of Celery in a mission critical situation?
Anything using redis will be riskier than AMQP by default.
Redis can be tuned to be safer (such as persisting on every key change) but this all comes with tradeoffs.
Redis such a great piece of tech I use it first as often as I can.
Re: Celery in production: Three more years of fixing bugs
#59I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
What have you switched to?
Re: Celery in production: Three more years of fixing bugs
#60Earlier quoted context omitted.
Others above have been suggesting Huey as an alternative. Do you know of any reason that it falls short of Celery in a mission critical situation?
Anything using redis will be riskier than AMQP by default. Redis can be tuned to be safer (such as persisting on every key change) but this all comes with tradeoffs. Redis such a great piece of tech I use it first as often as I can.
Sorry why is Redis riskier?