Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Celery 4.0

docs.celeryproject.org

Post reply on HN
Print view View original

11–20 of 51 posts

Re: Celery 4.0

#11

Post by andybak »

parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years.

I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that are more than good enough for most.

Re: Celery 4.0

#12

Post by ergo14 »

asksol wrote: post #10
ben_jones wrote: post #2

IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding. Edit: Apparently they curtailed some features for simplicity as well.

Thank you for the kind words, it's so very appreciated :) I have merged many features, like broker transports, result backends, etc, and while the initial contribution was great, it ends up being unmaintained with issues that nobody fixes. If there's any feature that you really want back, chances are the problems with that feature are not super difficult to fix, so please reach out!

Asksol we use celery at AppEnlight extensively and love it. Thank you for your great work.

Re: Celery 4.0

#13

Post by asksol »

andybak wrote: post #11
parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production.

Disclaimer: I'm a contributor

Re: Celery 4.0

#14

Post by andybak »

asksol wrote: post #13
andybak wrote: post #11

Earlier quoted context omitted.

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production. Disclaimer: I'm a contributor

I'm not saying Celery is over-engineered in general. It's just over-engineered in the context I've often seen it recommended in. i.e. for people learning or people with fairly modest requirements.

Re: Celery 4.0

#15

Post by fuhrysteve »

Congratulations to @asksol and the rest of the team for sealing the deal on 4.0! I've been waiting for a number of the features now in 4.0 for a long time now. I know you guys have been busting your asses for a long time and juggling with some complicated dependencies along the way.

Some time ago I built a little celery addon library as a sort of experimental way to solve the problem of having dynamic celery beat scheduled tasks. I never ended up implementing anywhere for a few different reasons: https://github.com/fuhrysteve/CeleryStore

I really like the concept behind the old djcelery project. But I don't use django much these days, and I'd like for it to be more compatible with tools I've become more familiar with (sqlalchemy / etc).

Do you have any advice for how to approach this? I know 4.0 introduced some new abilities to add beat entries via API.

Re: Celery 4.0

#16

Post by k_sze »

asksol wrote: post #13
andybak wrote: post #11

Earlier quoted context omitted.

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

Celery was never meant as a replacement for cron, it was simply a nice bonus that fits the messaging pattern well. Writing a task queue is actually very simple using for example Redis, but that doesn't necessarily mean Celery is over-engineered IMHO. It's very easy to forget the support required once your system is in production. Disclaimer: I'm a contributor

By "contributor" you mean you are the main author, right? XD

Joking aside, we use Celery in production for asynchronous number crunching. We build a web UI in Django that raises Celery tasks that run long number crunching jobs. When the number crunching is done we look at the report through our Django web UI.

It works well enough, though we wish there were better built-in task management. E.g. A built-in API to reshuffle tasks (e.g. for I/O resource balancing) would be nice. celery purge also seems too drastic in purging every task in every queue. Or maybe we're just doing it wrong.

Re: Celery 4.0

#17

Post by rtpg »

andybak wrote: post #11
parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

I think the advantage with tools like celery is that it deals with the many different failure scenarios pretty well.

It's kind of like jQuery's ajax. Of course you can figure out how to make a simple replacement. But then you have to manage the 100 different edge cases for when your code doesn't go down the happy path.

Much easier to write a simple task queue that's good enough than a replacement for $.ajax, though...

Re: Celery 4.0

#18

Post by stuaxo »

On my last project I had to use Celery. In the end, I had to look inside the source... having done the same with Python + Django I was apprehensive.

In the end I shouldn’t have been, it was a pleasant surprise. It's great 4.0 has come out.

Almost every Django developer will touch celery at some time, if your organisation can support development please do, it's not just an important piece of software, but well put together too.

Re: Celery 4.0

#19

Post by ceronman »

"Starting from Celery 5.0 only Python 3.5+ will be supported."

This is a small detail, but I'm glad that one more big Python project is dropping 2.x support. The 2/3 split is not good for the community.

Re: Celery 4.0

#20

Post by gcb0 »

a rabbitMQ client? what's so awesome that everyone here says it is essential for python web projects? Honest question, never used python for that.
Post reply on HN