Asynchronous Tasks with FastAPI and Celery

I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305

I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".

Re: Asynchronous Tasks with FastAPI and Celery

#5

Post by techhazard » Wed, May 12, 2021, 9:30 AM UTC

Personally I prefer meesee[1], because setting up Celery or other big worker systems is always a lot of work, and they have more components than I usually need.

Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you.

You just need a redis instance and then you're good to go :)

1: https://github.com/Attumm/meesee

Re: Asynchronous Tasks with FastAPI and Celery

#6

Post by scrollaway » Wed, May 12, 2021, 9:48 AM UTC

techhazard wrote: post #5

Personally I prefer meesee[1], because setting up Celery or other big worker systems is always a lot of work, and they have more components than I usually need. Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you. You just need a redis instance and then you're good to go :) 1: http…

Sounds very similar to RQ. Do you have opinions on the latter and/or their differences?

Re: Asynchronous Tasks with FastAPI and Celery

#7

Post by techhazard » Wed, May 12, 2021, 9:57 AM UTC

scrollaway wrote: post #6

techhazard wrote: post #5

Personally I prefer meesee[1], because setting up Celery or other big worker systems is always a lot of work, and they have more components than I usually need. Meesee is an elegantly simple worker system (written in python) using redis as a backend, and it works really well if you don't need all those extra features that stuff like Celery give you. You just need a redis instance and then you're good to go :) 1: http…

Sounds very similar to RQ. Do you have opinions on the latter and/or their differences?

I haven't heard of RQ before, but on first glance they look pretty similar.

The main difference I see is that meesee workers are always setup as python programs, whereas RQ workers are separate programs. (calling a `rq` program insead of `python3 my_worker.py`) Though looking at the docs tells me that RQ can do the latter as well.

Re: Asynchronous Tasks with FastAPI and Celery

#8

Post by ForHackernews » Wed, May 12, 2021, 2:13 PM UTC

_ZeD_ wrote: post #3

Isn't this basically the core of airflow?

Airflow is painful if you want to use it for anything other than time-bounded batch jobs. See https://medium.com/the-prefect-blog/why-not-airflow-4cfa4232...

I have no financial interest in Prefect, but I have used Airflow and their criticisms ring true.

Re: Asynchronous Tasks with FastAPI and Celery

#9

Post by staticautomatic » Wed, May 12, 2021, 2:17 PM UTC

svvvy wrote: post #4

I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305 I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".

I love Dramatiq but I don’t get your point about task completion. You have to choose between at most and at least once.

Re: Asynchronous Tasks with FastAPI and Celery

#10

Post by nicolaslem » Wed, May 12, 2021, 3:59 PM UTC

staticautomatic wrote: post #9

svvvy wrote: post #4

I have found Celery to be a major pain. It's very complicated (Are your Celery tasks guaranteed at-most-once or at-least-once delivery? Are you sure?). There are bugs with the Redis broker that cause it to drop tasks: https://github.com/celery/kombu/issues/305 I heartily recommend: https://dramatiq.io/motivation.html - "if you’ve ever had to use Celery in anger, Dramatiq could be the tool for you".

I love Dramatiq but I don’t get your point about task completion. You have to choose between at most and at least once.

The main problem with Celery is that it supports many brokers and many features. This creates many subtle edge cases, specifically for brokers not built with AMQP, for instance the "visibility_timeout" attribute of the redis broker which feels like a hack.

This is why I created yet another task framework based on redis with robustness as a the main goal: https://github.com/NicolasLM/spinach