Celery 4.0
docs.celeryproject.org
Celery 4.0
1–10 of 51 posts
Re: Celery 4.0
#2IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding.
Edit: Apparently they curtailed some features for simplicity as well.
Re: Celery 4.0
#3Just started using this for the first time. It's pretty nice. I'm still trying to figure out how to really use it with Flask/Connexion cleanly and not in the top level module, but it's made my life much simpler for handling long running tasks (e.g. When needing a status 202 response). Great project. Sad to see SQLAlchemy was removed from the brokers though I understand.
Re: Celery 4.0
#4Just started using this for the first time. It's pretty nice. I'm still trying to figure out how to really use it with Flask/Connexion cleanly and not in the top level module, but it's made my life much simpler for handling long running tasks (e.g. When needing a status 202 response). Great project. Sad to see SQLAlchemy was removed from the brokers though I understand.
You don't want to use a DB a broker.
For AWS they have SQS which is awesome. Now if someone would write a Google cloud pub/sub, all would be well in the world.
Re: Celery 4.0
#5IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding. Edit: Apparently they curtailed some features for simplicity as well.
Better to have 10 good features vs 20 bleh ones right?
Re: Celery 4.0
#6Just started using this for the first time. It's pretty nice. I'm still trying to figure out how to really use it with Flask/Connexion cleanly and not in the top level module, but it's made my life much simpler for handling long running tasks (e.g. When needing a status 202 response). Great project. Sad to see SQLAlchemy was removed from the brokers though I understand.
You don't want to use a DB a broker. For AWS they have SQS which is awesome. Now if someone would write a Google cloud pub/sub, all would be well in the world.
Hum, Google Pub/Sub seems roughly equivalent to SQS to me?
Re: Celery 4.0
#7IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding. Edit: Apparently they curtailed some features for simplicity as well.
Some have the opinion that Celery has (or had) too many features. I know I had to twiddle with a lot of settings in Celery for my unusual use case, but it worked well in the end. There was a long standing bug which ended up being my misunderstanding a certain setting. All that said, I would use Celery again.
Re: Celery 4.0
#8Celery is one of those things in Python that you can't (sometimes unfortunately) live without. Earlier versions of Celery have some difficult bugs and inconsistencies that made it feel like a very tough tool to work with and required a lot of developer diligence and operational experience to make sure your tool didn't break. Things like message memory explosion (multi-pass deserialization), poor defaults, difficulties debugging and tracking exceptions, poor monitoring tools like 'flower' etc. all lead to this. Problems were exacerbated by the fact that simple async operations in python (which are easily fixed in more concurrent languages with a simple go func{}()) end up requiring a heavy distributed solution like Celery (or the lighter RQWorker) which creates a whole host of issues.
I imagine as native async tooling improves in py3x (async/await, aiohttp, and other tools) use of celery to do trivially concurrent things will decrease and Celery's usage will focus on more complex workflows (chords, fanouts, map/reduce).
Looks like many concerns we're tackled here (thanks Celery team) and I'm looking forward to playing around with this release.
Re: Celery 4.0
#9Earlier quoted context omitted.
You don't want to use a DB a broker. For AWS they have SQS which is awesome. Now if someone would write a Google cloud pub/sub, all would be well in the world.
Hum, Google Pub/Sub seems roughly equivalent to SQS to me? https://cloud.google.com/pubsub/docs/overview
Yes but no one wrote the kombu broker for it yet to my knowledge. To use a new broker in Celery you have to write code on how to send and receive messages.
Re: Celery 4.0
#10IMO the secret ingredient to scaling almost any Python web application, or even implementing it in the first place. It's a shame they had to curtail some features due to lack of funding. Edit: Apparently they curtailed some features for simplicity as well.
Thank you for the kind words, it's so very appreciated :)
I have merged many features, like broker transports, result backends, etc, and while the initial contribution was great, it ends up being unmaintained with issues that nobody fixes.
If there's any feature that you really want back, chances are the problems with that feature are not super difficult to fix, so please reach out!