Live data from Hacker News

Celery in production: Three more years of fixing bugs

ayushshanker.com

61–63 of 63 posts

Re: Celery in production: Three more years of fixing bugs

#61

Earlier quoted context omitted.

Anything using redis will be riskier than AMQP by default. Redis can be tuned to be safer (such as persisting on every key change) but this all comes with tradeoffs. Redis such a great piece of tech I use it first as often as I can.

Sorry why is Redis riskier?

Because it's default configuration and (most) naive setups offer no redundancy? It is almost entirely in-memory with no easy eviction controls? If you are going to snapshot on every change because the thing you are doing needs to happen another purpose built queue like rabbitmq is going to perform better in this situation, especially during failures.

Clustering is another topic entirely.

Re: Celery in production: Three more years of fixing bugs

#62
post #3

I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.

IMO using python at all for this kinda setup with lots of workers is kinda crazy. As there is no usable threading, each worker need its own process. And as the article says, that's a couple of hundred mb of ram per worker. But in other languages, you could have hundreds of threads doing work with the same resources.

You can have async workers or gevent workers in python.

Re: Celery in production: Three more years of fixing bugs

#63

I wrote an extremely performant and simple async worker framework called SAQ because I couldn't find any that fit my use case. https://github.com/tobymao/saq

You said you drew inspiration from arq. What was lacking with arq that made you want to write something new?
Post reply on HN