An introduction to RabbitMQ

If the library were being designed from scratch today, pretty much every method on the model would be Async. After all, if it leads to any network I/O of any kind, that can block.

Working with the current public API, Trying to implement a publish wrapper that never blocks, and returns a task that either completes when the publisher confirm is received, or faults after some provided timeout, is a lot trickier than it might sound.

Recovery from network interruptions is complicated, and auto-recovery features are limited, and in some use cases actually dangerous. For example, if you are manually acknowledging messages to ensure end-to-end at-least-once delivery, then you cannot safely use the auto-recovery, since the delivery numbers would reset when the connection does, and you can accidentally aknowlodge the wrong message with delivery tag 5. (Acknowledge the new one, when you were trying to ack the old one).

In my implementation of that included my own recovery, I ended up needing to pass around the IModel itself with the delivery tags, so I can check if the channel I am about to acknowledge on is really the same one I received the message on. (There is no unique identifier of a channel instance, since even the channel number is likely to get re-used).

I can offer a similar anecdote: we started seeing rabbitmq reporting alleged cluster partitions in production after enabling TLS between rabbitmq nodes, where manual recovery was needed each time.

After a bit of investigation we noticed that cluster partition seemed to correlate with sending an unusually large message (think something dumb like 30 megs) through rabbitmq when TLS between rabbitmq nodes was enabled. What I believe was happening was Rabbitmq was so busy encrypting/decrypting large message that it delayed sending or receiving heartbeat & then the cluster falsely assumed there has been a network partition.

Mitigated that issue by rewriting system to not send 30 meg messages- there was only one message producer that sent messages anywhere near that large, and after a bit of thought realised it was not necessary to send any message at all in that case (sending large message was to hack around some other old system performance problem that had gotten fixed properly a year back, but the hack that generated a huge message was still in place)

When I first started using RabbitMQ I experienced just about everything you described.

I felt incredibly stupid when a customer would have issues with a queue being stuck or messages that were being dropped, and having no clue on why this was happening.

> It's nearly impossible to get RabbitMQ working correctly within few months.

This is so true. You can get it running in 10 minutes, but it takes weeks of banging your head against the wall and angry customers before you have it running right.

A common use-case is for async tasks (Celery) that can tolerate a few minutes of downtime. If you're running a fully evented architecture then this might not apply - though if you're not targeting 4-5 nines of reliability or an RTO of For context I had an RMQ docker container running for almost two years without any interruptions. If you're in a small team then HA might well be overkill.

Fun gotcha - if you're running RMQ in Kubernetes/Docker, make sure you give it an explicit memory limit, else it will try to allocate disk space equal to 40% of your host's memory. (See "memory limits" in https://hub.docker.com/_/rabbitmq). That's a good best-practice for any containerized environment regardless what workload you're running, but this one will cause errors if you're trying to use a small disk volume on a host with lots of memory.