Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Celery in production: Three more years of fixing bugs

ayushshanker.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Celery in production: Three more years of fixing bugs

#31

Post by postpawl »

alesso_x wrote: post #21
tluyben2 wrote: post #3

I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.

I totally agree, I would not recommend using Celery. The last time I used Celery was in 2019 and it was a mess. The memory leak issue is still open https://github.com/celery/celery/issues/4843 to this day.

There has been a lot of progress on fixing memory leaks recently: https://www.paulsprogrammingnotes.com/2021/12/python-memory-...

Re: Celery in production: Three more years of fixing bugs

#32

Post by true_religion »

My advise is to just pay someone to run your celery server for you.

Celery is hard to maintain, but the benefits are definitely worth it, and it’s the most mature cross language stack. I can have a celery job sent or received from any major language.

I think the only reason people pick another option is that that start of by only using python in the first place.

Re: Celery in production: Three more years of fixing bugs

#33

Post by jhgg »

Celery is not great. At my place of work we begrudgingly are using it right now, with plans to replace it with something else this year. I am at the very least amazed we got this much use out of it. Considering replacing it with a much much more simple version that just uses google cloud pub sub and some simple python routing of the messages to functions.

Re: Celery in production: Three more years of fixing bugs

#34

Post by tux1968 »

leetrout wrote: post #23
gilad wrote: post #13

RQ [0] (Python-based with Redis backend) is another option; works great. [0] https://python-rq.org/

I love RQ. Nothing but good experiences using it several times over the past ~6 years. However if you are doin mission critical work in your queue that cannot be replayed Celery is still the best python flavored option IMO

Others above have been suggesting Huey as an alternative. Do you know of any reason that it falls short of Celery in a mission critical situation?

Re: Celery in production: Three more years of fixing bugs

#36

Post by andrewstuart »

I suspect in many cases people use Celery only to send emails or for simple queueing type problems.

At risk of being overly negative, my experience with celery has been that it is complex and hard to configure and debug.

I chose a simple queueing mechanism instead of using Celery, and for sending emails I was so frustrated by Celery that I wrote the Arnie SMTP buffer server (in 100 lines of Python): https://github.com/bootrino/arniesmtpbufferserver

Re: Celery in production: Three more years of fixing bugs

#37

Post by sergiomattei »

loloquwowndueo wrote: post #17
js2 wrote: post #15

Earlier quoted context omitted.

I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.

Reddit :)

Did we just witness a Freudian slip? :)

Re: Celery in production: Three more years of fixing bugs

#38

Post by crucialfelix »

js2 wrote: post #15
tluyben2 wrote: post #3

I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.

I've been running four separate installs of Celery since 2014 or so processing about 10-20 million jobs a day, using reddit as a broker, and with about 100 worker instances per install. It just works, but I don't use Celery results or any esoteric features.

I used it for 10 years myself. Also learned the hard way not to use esoteric features.

I did have to build my own solutions for things like priority queues, but it was much clearer and testable.

Re: Celery in production: Three more years of fixing bugs

#39

Post by sillycube »

tluyben2 wrote: post #3

I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.

Do you have any suggestions for alternatives?

Re: Celery in production: Three more years of fixing bugs

#40

Post by porker »

gregsadetsky wrote: post #7
tluyben2 wrote: post #3

I ran Celery in many projects in production over the past 10 years; I would not recommend it. It is mostly a constant fight even though it is the first thing most people grab when they have a Django stack.

What would you recommend instead?

Beanstalkd?

These comments say better than I can why I like it

https://news.ycombinator.com/item?id=27483167

https://news.ycombinator.com/item?id=24714198

Post reply on HN