Celery in production: Three more years of fixing bugs

I totally agree, I would not recommend using Celery. The last time I used Celery was in 2019 and it was a mess. The memory leak issue is still open https://github.com/celery/celery/issues/4843 to this day.

I love RQ. Nothing but good experiences using it several times over the past ~6 years.

However if you are doin mission critical work in your queue that cannot be replayed Celery is still the best python flavored option IMO

I’ve been using Celery and it has been working okay for our needs but it is not very friendly to make changes depending on how you write tasks. I feel okay when a task is a wrapper around some other functions that you can just run and test with any tools you’d like but we have some tasks that go all in by binding all kids of data and functionality to the task that isn’t guaranteed to be there if you just launch a task with .delay or similar so you can’t just spin up a REPL or debugger and call the function and have it work.

Even if there were real threading, it has features that make it convenient for scheduling tasks, chaining them in different ways and so on. You could write your own thing of course. But also it operates over different processes and those processes don't even necessarily have to be on the same machine.

Task queue pattern is not unique to python.

Consider where you see physical queues of people in real world: e.g. people lining up to be served at supermarket, to board aircraft, to enter a concert. The purpose of the queue is an orderly way to manage situations where there is more demand for a service than can be immediately processed. If the system tries to serve all the demand simultaneously, perhaps that leads to chaos and no one getting served.

Similar situations happen in computer systems, in some of those cases queues can be used to manage load.

E.g. When you make a purchase to order something online, there is a good chance there is a task queue or something very similar as part of the machinery helping that to happen. When you click "purchase" your order or request is persisted in some queue or DB, before any attempt is made to fulfill it. The UI can give you immediate feedback telling you that your order was received, here's your order ID, sit back and relax, we'll send you an update. When some machine (or person) is available with free capacity, and no higher priority orders to process, they will grab your order and start processing it.

Some worker queue implementations offer additional benefits in terms of increasing robustness of system. e.g. if worker node A begins processing task T, then worker node A explodes before T is complete, the task T may be returned to the queue after some timeout, and surviving worker B may be able to grab T and process it successfully. This can be very valuable in situations where you want to guarantee that task T is done correctly and not half-done or forgotten -- e.g. processing and provisioning customer orders. This gets considerably more challenging to implement correctly if task T involves performing side effects and modifying the state of the real world (transferring money, dispatching vehicles) and the failure occurs after some but not all of the side effects have already happened.

Queues introduce new failure modes: if something stops workers from processing tasks, and queue gets longer and longer, how do you find out and fix it? Queue needs to be stored somewhere, there is some max capacity, what happens when that is exceeded -- does the queue give up and drop all the tasks on the floor, or explode, or so on.