The Many Problems with Celery

#1

Post by sbdchd » Sun, May 21, 2023, 3:21 PM UTC

The Many Problems with Celery
steve.dignam.xyz

Re: The Many Problems with Celery

#2

Post by Something1234 » Mon, May 22, 2023, 1:13 AM UTC

Celery has so many issues and this outlines just a few of them. I was working on celery for a while on one of my personal projects and it's just absolutely insane what you have to do to get it working with django. It's so incredibly over engineered nad powered by magic.

Re: The Many Problems with Celery

#3

Post by dikei » Mon, May 22, 2023, 1:57 AM UTC

I think Celery is doing itself a disservice by supporting so many brokers and result stores with different behaviors about delivery guarantee, leading to leaky abstraction everywhere.

Re: The Many Problems with Celery

#4

Post by iancmceachern » Mon, May 22, 2023, 2:14 AM UTC

I came here to air my grievances with the vegetable. Alas, not tonight.

Re: The Many Problems with Celery

#5

Post by z3ugma » Mon, May 22, 2023, 12:42 PM UTC

For folks who have tried celery, and were dissatisfied with that, what did you switch to? What else is there for python task queues?

Re: The Many Problems with Celery

#6

Post by 1letterunixname » Mon, May 22, 2023, 4:30 PM UTC

iancmceachern wrote: post #4

I came here to air my grievances with the vegetable. Alas, not tonight.

Have you tried it with peanut butter? It causes all jobs to work swimmingly.

Re: The Many Problems with Celery

#7

Post by 1letterunixname » Mon, May 22, 2023, 4:40 PM UTC

There should be a distinction between a work queue and a job scheduler. The work queue dispatches jobs but doesn't know enough to retry them while the scheduler decides what to retry.

In Rails land, I would opt for Backburner over the commercial choices because it uses beanstalkd. For anything requiring HA, I would look at some sort of queue using an ESB as a transport. Short of HA, shard for LB.

Re: The Many Problems with Celery

#8

Post by Beestie » Mon, May 22, 2023, 4:42 PM UTC

Celery? Not knowing this was a post about software, I figured it was about the vegatable.

Can't wait until software naming reaches the same absurdity level as racehorse names.

/rant

Re: The Many Problems with Celery

#9

Post by shaftway » Mon, May 22, 2023, 8:49 PM UTC

1letterunixname wrote: post #6

iancmceachern wrote: post #4

I came here to air my grievances with the vegetable. Alas, not tonight.

Have you tried it with peanut butter? It causes all jobs to work swimmingly.

The problem is that it's difficult to add raisins without them being consumed sequentially.

Re: The Many Problems with Celery

#10

Post by iancmceachern » Tue, May 23, 2023, 3:26 AM UTC

shaftway wrote: post #9

1letterunixname wrote: post #6

Earlier quoted context omitted.

Have you tried it with peanut butter? It causes all jobs to work swimmingly.

The problem is that it's difficult to add raisins without them being consumed sequentially.

The problem in my view is those tooth floss like strings. They're worse with peanut butter. If one could figure out celery without those terrible strings I'd happily reconsider my opinion on celery.