41–50 of 63 posts
We got a lot of mileage out of it (in a past life) before finally moving our job scheduling to a custom solution built on RQ. Celery caused more than a few headaches -- which is why we ditched it -- but its flexibility probably helped us scale up the service to begin with...
Can any explain why we need task queue? It is unique to Python because of it lack of real threading?
For example imagine you want to receive a file from the user and extract data from it.
Using such system also allowed us to horizontally scale by adding more worker machines.
All this, of course, depends on the tasks you want to schedule.
Earlier quoted context omitted.
What would you recommend instead?
Beanstalkd? These comments say better than I can why I like it https://news.ycombinator.com/item?id=27483167 https://news.ycombinator.com/item?id=24714198
And yes, celery also supports beanstalk, so it's clearly not an alternative to it
http://docs.celeryproject.org/en/latest/getting-started/brok...
I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
What would you recommend instead?
It has its own set of problems (mostly from starting at the set of "we will remove all of Celery features" and then going from there), in particular its logging policies are extremely frustrating and we have had to work around them. But the code is so simple in general that we get good compensation for that.
I personally would have liked to use dq instead but that's mostly an aesthetics thing.
Does anyone have any experience with it? I like the pessimistic execution concept..
I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
What would you recommend instead?
I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.
What would you recommend instead?
Earlier quoted context omitted.
I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.
Reddit :)