An introduction to RabbitMQ

If you're bad at hosting and need the throughput, there's cloudamqp.

So many options for pub/sub systems so use what works for you.

I wish a standard set of higher abstractions existed on it though. Celery, from what I hear fills that gap very well in the python world but nothing like this exists in Nodejs land which leaves the room open for a bunch of redis-backed solutions which are pretty fragile in comparison.

Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika library was replaced by 3 lines of python and redis server.

One day rabbitmq died and it tooks sys admins few weeks to get it back running. In that time I deployed my simple redis list and never looked back. To this day redis solution works without any friction whatsoever with fraction of resources.

The rabbits AMQP exchange model is severely flawed and convoluted. It's the worst example of corporate software where everything works and doesn't work at the same time.

I wouldn't recommend rabbitmq to my worst enemy yet there's still something attractive about it. Maybe there's a sane alternative? Maybe zeromq?

My experiences were pretty mixed. Overall I found it to be more difficult than I would have wanted to get simple things to work. Part of this seems to be a problem with the Java library, which is not great. For example, IIRC you have to be really careful not to create the same queue twice, even with identical configurations, since the second time something blows up. At the end of the day just a simple fan-out configuration ends up involving a lot of somewhat-intricate code. It definitely does not Just Work (TM).

And then there was the bizarre hangs that I would experience during testing. I set up a Docker Compose configuration so that I could test the various parts of the system independently. It included one container running RabbitMQ to simulate the cluster we have running on our cloud.

Usually tests ran fine. But then, from time to time, the client would just hang trying to send a message through RabbitMQ. Unfortunately, again, the code you need to just run a basic configuration using RabbitMQ is complex enough that at first I was pretty sure that I had done something wrong. But after a few hours of increasing frustration I finally broke down and discovered that a simple test case that just sent a single message using code torn right out of the docs would hang. Forever. (Or, long enough that I gave up waiting.)

After a lot of digging I found the culprit. RabbitMQ will just take its ball and go home if the broker doesn't have enough disk space. Given that I use Docker heavily for a lot of projects, the amount available to new containers would vary a lot depending on what other data sets I had loaded or how recently I had run docker system prune.

I filed an issue about this, asking to have a better error message displayed when an attempt to send a message was made. The response was: there's already an error message, printed during startup. You didn't see it? No. I must have missed it among the hundreds of other lines of output that RabbitMQ spews when it starts.

Overall my favorite part of this story is that RabbitMQ chooses to start but refuse to send messages when low on disk space, when just crashing would be much more useful and make it much easier to pinpoint what was going on.

Anyway, I'm in the market for a simpler alternative that's Kotlin friendly.