Show HN: Dramatiq – An alternative to Celery

It doesn't, but you're the second person to request it. Feel free to open an issue and I'll take a stab at writing an SQS broker. Having looked at it before, it didn't seem like it would take much work at all.

> I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens.

Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core.

> If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.

I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)

Seriously? RabbitMQ is a huge waste of complexity for like 95% of modern webapp use cases.

I use tools like Dramatiq specifically because I want to abstract away the complexity of exchanges, bindings, queues, dead lettering, etc...

If you write out all the nouns associated with RMQ you start to realize there is a LOT going on. It’s a very powerful system but convention over configuration doesn’t work here.

I’ve submitted code to this project and just want to state here for the record that Bogdan is very responsive and accepting of contributions. It was a very pleasing experience compared to other open source projects!

I reached out to the Python community recently with the question: “Django is to Flask as Celery is to ______?” Dramatiq was one of the suggested responses and the mission/purpose behind it resonated with me immediately.

I’m a big fan of convention over configuration. Most of the time, if I’m writing software in Python, I don’t want to worry about a system that is not Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it abstracts Rabbit. Obviously you need to understand the underlying system and an ORM is no replacement for knowing the ins and outs of Postgres, just as this is no replacement for understanding Rabbit. But... I love that it does everything I want it to do with minimal or zero configuration.

Thanks so much. You're right, I think some people have written storage engines for rabbit (as well as mongo) but you might have to search for them. The storage api is simple enough you could write your own in an hour or two.