Celery 4.0 • Hacker News Board

Re: Celery 4.0

#21

Post by jjawssd » Sun, Nov 13, 2016, 3:38 AM UTC

rtpg wrote: post #17

andybak wrote: post #11

Earlier quoted context omitted.

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

I think the advantage with tools like celery is that it deals with the many different failure scenarios pretty well. It's kind of like jQuery's ajax. Of course you can figure out how to make a simple replacement. But then you have to manage the 100 different edge cases for when your code doesn't go down the happy path. Much easier to write a simple task queue that's good enough than a replacement for $.ajax, though..…

Well said

Re: Celery 4.0

#22

Post by tschellenbach » Sun, Nov 13, 2016, 3:41 AM UTC

Many thanks Asksol, beautiful software. Have been a happy user for many years now. Hope to try out 4.0 soon!

Re: Celery 4.0

#23

Post by rch » Sun, Nov 13, 2016, 3:54 AM UTC

parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

For little things, it doesn't matter much what you do, and for bigger things I've moved to Luigi for background tasks.

Re: Celery 4.0

#24

Post by cwyers » Sun, Nov 13, 2016, 5:11 AM UTC

> Nowadays it’s easy to use the requests module to write webhook tasks manually. We would love to use requests but we are simply unable to as there’s a very vocal ‘anti-dependency’ mob in the Python community

I'm not a heavy Python user, and I've never heard this before. It sounds... less than good.

Re: Celery 4.0

#25

Post by gtaylor » Sun, Nov 13, 2016, 6:21 AM UTC

andybak wrote: post #11

parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

For low and medium traffic sites, the simplest usage patterns are really easy: http://docs.celeryproject.org/en/latest/getting-started/intr...

The best thing about celery is that you don't have to use any of the advanced features. If you need the basics, stick to the basics.

On the flipside, if and when you do need more than the basics, your system will grow with you. No need to hack up cron monstrosities as you grow beyond a single server. Though, cron vs celery is kind of an apples to... celery comparison.

Re: Celery 4.0

#26

Post by gtaylor » Sun, Nov 13, 2016, 6:23 AM UTC

cwyers wrote: post #24

> Nowadays it’s easy to use the requests module to write webhook tasks manually. We would love to use requests but we are simply unable to as there’s a very vocal ‘anti-dependency’ mob in the Python community I'm not a heavy Python user, and I've never heard this before. It sounds... less than good.

It's not universal. I'd rather bring in a dependency than reinvent the wheel.

requests is used so widely that I'm surprised to see this raised as an issue.

Re: Celery 4.0

#27

Post by pandler » Sun, Nov 13, 2016, 6:24 AM UTC

gcb0 wrote: post #20

a rabbitMQ client? what's so awesome that everyone here says it is essential for python web projects? Honest question, never used python for that.

I haven't used Django in a while, but, if I'm not mistaken, it's basically the easiest way to add any kind of asynchronous behavior to a python web server. There are few other options, and I don't think any of them are as mature as Celery.

Re: Celery 4.0

#28

Post by sandGorgon » Sun, Nov 13, 2016, 6:57 AM UTC

Has anybody compared Celery 4 to rq (which started off as a much simpler+performant alternative to celery).

Would love to see how they stack up after this release.

Re: Celery 4.0

#29

Post by hueving » Sun, Nov 13, 2016, 7:28 AM UTC

asksol wrote: post #10

ben_jones wrote: post #2

IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding. Edit: Apparently they curtailed some features for simplicity as well.

Thank you for the kind words, it's so very appreciated :) I have merged many features, like broker transports, result backends, etc, and while the initial contribution was great, it ends up being unmaintained with issues that nobody fixes. If there's any feature that you really want back, chances are the problems with that feature are not super difficult to fix, so please reach out!

Do you have a low-hanging fruit tag for new contributors to start with?

Re: Celery 4.0

#30

Post by asadjb » Sun, Nov 13, 2016, 7:38 AM UTC

sandGorgon wrote: post #28

Has anybody compared Celery 4 to rq (which started off as a much simpler+performant alternative to celery). Would love to see how they stack up after this release.

We recently moved one of our services from Rq to Celery. We are using an older version of Celery, but this comment should still apply. While Rq is a great way to go in the start, if you have a large number of messages you are handling, you'll get into memory bottlenecks. This isn't a problem with Rq, but with using Redis as a broker. So if anyone is considering Rq vs Celery, they should keep this in mind.

The reason we switched to Celery was the volume of messages we were handling. Since Rq relies on Redis, all your messages need to fit in memory. While Rq was great and simple to setup at start, as we grew we were consistently dealing with Rq breaking because Redis was full and stopped accepting any write operations.

We moved to Celery because it could use RabbitMQ as a broker. RabbitMQ offloads most messages to the disk which has nicely taken care of the memory limitation issues.

With Rq we would get stuck after 10K messages (our messages included images so individual message size was large). With RabbitMQ I've seen the queue grow to about 120K without so much as a single hiccup.