An introduction to RabbitMQ • Hacker News Board

Re: An introduction to RabbitMQ

#191

Post by adamkf » Fri, May 22, 2020, 1:57 AM UTC

I've had truly terrible experiences with RabbitMQ. I believe that it should not be used in any application where message loss is not acceptable. Its two big problems are that it cannot tolerate network partitions (reason enough to never use it in production systems, see https://twitter.com/antifuchs/status/735628465924243456), and it provides no backpressure to producers when it starts running out of memory.

In my last job, we used Rabbit to move about 15k messages per sec across about 2000 queues with 200 producers (which produced to all queues) and 2000 consumers (which each read from their own queues). Any time any of the consumers would slow down of fail, rabbit would run out of memory and crash, causing sitewide failure.

Additionally, Rabbit would invent network partitions out of thin air, which would cause it to lose messages, as when partitions are healed, all messages on an arbitrarily chosen side of the partition are discarded. (See https://aphyr.com/posts/315-jepsen-rabbitmq for more details about Rabbit's issues and some recommendations for running Rabbit, which sound worse than just using something else to me.)

We experimented with "high availability" mode, which caused the cluster to crash more frequently and lose more messages, "durability", which caused the cluster to crash more frequently and lose more messages, and trying to colocate all of our Rabbit nodes on the same rack (which did not fix the constant partitions, and caused us to totally fail when this rack lost power, as you'd expect.)

These are not theoretical problems. At one point, I spent an entire night fighting with this stupid thing alongside 4 other competent infrastructure engineers. The only long term solution that we found was to completely deprecate our use of Rabbit and use Kafka instead.

To anyone considering Rabbit, please reconsider! If you're OK with losing messages, then simply making an asynchronous fire-and-forget RPC directly to the relevant consumers may be a better solution for you, since at least there isn't more infrastructure to maintain.

Re: An introduction to RabbitMQ

#192

Post by fmorel » Fri, May 22, 2020, 2:18 AM UTC

hnrodey wrote: post #60

I'm very well versed on RabbitMQ. We use it internally in a .NET codebase. Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some late…

If you're using RabbitMQ on .Net, I highly recommend using NserviceBus. It's made working with queues so easy. It handles maintaining a connection and retrying/acknowledging messages for you.

Re: An introduction to RabbitMQ

#193

Post by [deleted] » Fri, May 22, 2020, 2:26 AM UTC

polygotdomain wrote: post #147

hnrodey wrote: post #127

Earlier quoted context omitted.

>Can you talk a little bit about how you've managed your RabbitMQ infrastructure? From a ten thousand foot view, two or three node clusters running in non-prod environments on virtual machines running Windows. In Prod, three node clusters on Windows virtual machines. All work to install and configure RabbitMQ is done manually. Sadly enough. I'm on the application/architecture side of this equation but I know enough a…

Thanks for the response. From the other responses in this thread, it seems like the admin of the nodes/cluster is not overly onerous. Would you agree with that statement? Also, being a .Net shop, the Windows VMs make sense, but is there any tradeoffs to running Rabbit on Windows, as opposed to Linux? I think part of the sell is how we would manage the admin component of a Message Queue, which tilts things towards Azu…

[deleted]

Re: An introduction to RabbitMQ

#194

Post by Teknoman117 » Fri, May 22, 2020, 2:36 AM UTC

adamkf wrote: post #191

I've had truly terrible experiences with RabbitMQ. I believe that it should not be used in any application where message loss is not acceptable. Its two big problems are that it cannot tolerate network partitions (reason enough to never use it in production systems, see https://twitter.com/antifuchs/status/735628465924243456 ), and it provides no backpressure to producers when it starts running out of memory. In my l…

We had a similar experience where I work. We just ended up rolling our own queue system because we really just needed point to maybe a few other points.

Re: An introduction to RabbitMQ

#195

Post by m3nu » Fri, May 22, 2020, 2:59 AM UTC

mythrwy wrote: post #153

vasco wrote: post #144

Earlier quoted context omitted.

Celery can be backed by RabbitMQ, not sure if that's what you meant, but all of what you described can be abstracted away. I didn't have the same experiences with months taken to get up to speed. Moreover, at work RabbitMQ is probably our most stable underlying tool, perhaps toe to toe with Redis. And that's saying a lot, since I consider Redis to almost be a piece of art in how great of a tool it is. Back to RabbitM…

I have simple single node deployment and I was floored how easy it was to set up with Celery. Really surprised. I was kicking myself for not using it sooner. Granted I don't know all the intricacies of RabbitMQ and this was just one step beyond os.popen, but it was painless, like half an hour painless to set up and it has worked really well. *edit: reading some of the other posts now I'm waiting for the other shoe to…

Same experience. Single node with a few clients and Celery. Works well.

My main issue in the beginning were network timeouts now and then. Those went away after tuning some TCP settings.

Re: An introduction to RabbitMQ

#196

Post by keithnz » Fri, May 22, 2020, 3:12 AM UTC

anon176 wrote: post #170

keithnz wrote: post #167

Earlier quoted context omitted.

weird, I haven't done much digging in to the details of RabbitMQ, but I integrated it in a matter of hours, and have it deployed in production systems (for quite sometime now) and it works really solidly. I haven't tried to get too clever though.

You must of followed a good guide on getting it setup. Took me 2 days to get it solid. Then we decided to just use redis.

I just used the official docs and guides they had on the website, they seemed pretty good to me. I might have googled a few extra things, but can't really remember, I just remember it being pretty straightforward. I remember they pointed out a number of things you had to take care of.

Re: An introduction to RabbitMQ

#197

Post by CloudButWhy » Fri, May 22, 2020, 3:14 AM UTC

tankerdude wrote: post #172

vasco wrote: post #144

Earlier quoted context omitted.

Celery can be backed by RabbitMQ, not sure if that's what you meant, but all of what you described can be abstracted away. I didn't have the same experiences with months taken to get up to speed. Moreover, at work RabbitMQ is probably our most stable underlying tool, perhaps toe to toe with Redis. And that's saying a lot, since I consider Redis to almost be a piece of art in how great of a tool it is. Back to RabbitM…

You would think, until you get to a split brain issue. The master and failover lose connectivity, and they each then think they're the master. There's ways to repair it (and it has happened to me one total time in 4 years), but it does happen. I personally try to make my message processing idempotent for the worker to help alleviate these situations.

There is a reason why you're supposed to run an odd number of nodes so that you will hopefully have a majority in case of a failure.

Re: An introduction to RabbitMQ

#198

Post by sethammons » Fri, May 22, 2020, 3:37 AM UTC

adamkf wrote: post #191

I've had truly terrible experiences with RabbitMQ. I believe that it should not be used in any application where message loss is not acceptable. Its two big problems are that it cannot tolerate network partitions (reason enough to never use it in production systems, see https://twitter.com/antifuchs/status/735628465924243456 ), and it provides no backpressure to producers when it starts running out of memory. In my l…

Wow, that error message! Unless you are Google, network partitions are a thing. With CAP, you don’t get to choose CA.

Re: An introduction to RabbitMQ

#199

Post by jonathanoliver » Fri, May 22, 2020, 3:39 AM UTC

ies7 wrote: post #171

mcsoft wrote: post #145

Earlier quoted context omitted.

http://nats.io

I heard a lot of praise about Nats, but isn't it more like a kafka alternative? Someone new need to spend sometime grasping the stream concept.

NATS by itself is designed to be more of an always-on style queuing system (the term they use is "dial tone") but doesn't handle node failures by itself. If you're looking for a Kafka-flavored NATS, there's a new release I saw recently called LiftBridge that adds some durability to the NATS protocol.

Re: An introduction to RabbitMQ

#200

Post by sethammons » Fri, May 22, 2020, 3:47 AM UTC

wegs wrote: post #65

My general problem is that it's really hard to figure out which architecture is right for which system. There's a different architecture for: * one queue with billions of messages * a millions of queues with small numbers of messages per queue * many queues with many messages per queue There are also different topologies: * Anyone can send a message to anyone (O(n^2) queues) * One publisher with millions of subscribe…

Fantastic points. We needed millions of queues with millions of items with fair queueing and scheduled release of some items and immediate release of others. 10s of thousands of messages per second. We had to build our own.