Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Celery 4.0

docs.celeryproject.org

Post reply on HN
Print view View original

31–40 of 51 posts

Re: Celery 4.0

#31

Post by sandGorgon »

asadjb wrote: post #30
sandGorgon wrote: post #28

Has anybody compared Celery 4 to rq (which started off as a much simpler+performant alternative to celery). Would love to see how they stack up after this release.

We recently moved one of our services from Rq to Celery. We are using an older version of Celery, but this comment should still apply. While Rq is a great way to go in the start, if you have a large number of messages you are handling, you'll get into memory bottlenecks. This isn't a problem with Rq, but with using Redis as a broker. So if anyone is considering Rq vs Celery, they should keep this in mind. The reason…

That's very insightful. Quick question, since you are doing this in production - how are you serializing images into a message ? Base64 or something else.

Re: Celery 4.0

#32

Post by asadjb »

sandGorgon wrote: post #31
asadjb wrote: post #30

Earlier quoted context omitted.

We recently moved one of our services from Rq to Celery. We are using an older version of Celery, but this comment should still apply. While Rq is a great way to go in the start, if you have a large number of messages you are handling, you'll get into memory bottlenecks. This isn't a problem with Rq, but with using Redis as a broker. So if anyone is considering Rq vs Celery, they should keep this in mind. The reason…

That's very insightful. Quick question, since you are doing this in production - how are you serializing images into a message ? Base64 or something else.

We just do a str(file.read()). Not the most elegant solution. :)

Re: Celery 4.0

#33

Post by ryankask »

cwyers wrote: post #24

> Nowadays it’s easy to use the requests module to write webhook tasks manually. We would love to use requests but we are simply unable to as there’s a very vocal ‘anti-dependency’ mob in the Python community I'm not a heavy Python user, and I've never heard this before. It sounds... less than good.

The author may be referring to libraries themselves having dependencies. These attitudes are changing. For example, last week the Django community published a draft proposal that declares that "Django can have dependencies". The "Background" section is a good read and helps explain the origins of these attitudes: https://github.com/django/deps/blob/master/draft/0007-depend....

Re: Celery 4.0

#35

Post by andybak »

gtaylor wrote: post #25
andybak wrote: post #11

Earlier quoted context omitted.

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

For low and medium traffic sites, the simplest usage patterns are really easy: http://docs.celeryproject.org/en/latest/getting-started/intr... The best thing about celery is that you don't have to use any of the advanced features. If you need the basics, stick to the basics. On the flipside, if and when you do need more than the basics, your system will grow with you. No need to hack up cron monstrosities as you grow…

Even if the getting started is simple, there's still a ton of code and dependencies you're bringing into your project. Most importantly you've suddenly got dependencies on persistent processes such as Redis and RabbitMQ which need to be installed system wide. That's something that needs to be factored in to deploys, configs, restarts etc. You now need new tutorials for every deploy method (Heroku, Webfaction, PythonAnywhere and other outliers).

Yeah - everyone should ideally be comfortable with all this stuff but I try and keep anything that's more complex than a pip install in a virtualenv to an absolute minimum.

Re: Celery 4.0

#36

Post by zo1 »

sandGorgon wrote: post #31
asadjb wrote: post #30

Earlier quoted context omitted.

We recently moved one of our services from Rq to Celery. We are using an older version of Celery, but this comment should still apply. While Rq is a great way to go in the start, if you have a large number of messages you are handling, you'll get into memory bottlenecks. This isn't a problem with Rq, but with using Redis as a broker. So if anyone is considering Rq vs Celery, they should keep this in mind. The reason…

That's very insightful. Quick question, since you are doing this in production - how are you serializing images into a message ? Base64 or something else.

Not the person you're asking, but from the experiences I've had with Celery + other messaging queues: Don't pass around large binary blobs if you can avoid it. Whether that is an image or something else is irrelevant.

You can do such things as passing a database unique key, GUID, or file-path to the raw data on disk. Obviously, you will also need to engineer around that if you've got a distributed system. The tangential benefit is that you're not using a "messaging" queue or system for persisting or semi-persisting your image data. That's a big no-no as such systems are transient in nature and that often doesn't align with binary or image processing.

Base64 is for when you want to pass around binary in a text-based format. E.g. XML or JSON. But do keep in mind that because of the encoding format, converting binary data to base64 does increase the payload size by about 15% or so.

Re: Celery 4.0

#37

Post by zo1 »

pandler wrote: post #27
gcb0 wrote: post #20

a rabbitMQ client? what's so awesome that everyone here says it is essential for python web projects? Honest question, never used python for that.

I haven't used Django in a while, but, if I'm not mistaken, it's basically the easiest way to add any kind of asynchronous behavior to a python web server. There are few other options, and I don't think any of them are as mature as Celery.

The python Tornado web server does have the ability to do async tasks. However, it'd better be non-blocking otherwise your nice async web-server is going to stop serving requests while your task is processing.

Re: Celery 4.0

#38

Post by bar-b-q »

andybak wrote: post #11
parhamn wrote: post #8

Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficultie…

I've managed to live without it. For low and medium traffic sites it's hugely over-engineered. For all the sites I manage, I run a single cron task that triggers a range of background jobs. It's an approach worked very well for nearly 10 years. I understand there's a genuine use-case for Celery but like many technologies people are told "if you need a task queue use this" when there are much simpler solutions that ar…

When building a small web app a few weeks ago, I tried to avoid additional dependencies, too. I was a bit stuck when I need to send an email asynchronous. Do you have a _simple_ recommendation to solve this without pulling something huge like celery+redis in to solve this task?

Re: Celery 4.0

#39

Post by fuxi »

I have been using celery for years now and I love it. I was excited to see 4.0 with SQS support. However after giving it a spin I run back to 3.x.

Unfortunatelly version 4.0.0 has too many bugs for now. I will need to check back in few months. Better tests and better mocks would prevent the issues.

Re: Celery 4.0

#40

Post by sateesh »

logronoide wrote: post #34

Worst experience with a python toolkit ever. I hope new versions fix all bugs and issues.

Being a user of celery, I find this comment unfair.Adding some points about why you consider Celery to be bad will help.
Post reply on HN