An introduction to RabbitMQ • Hacker News Board

Re: An introduction to RabbitMQ

#111

Post by polygotdomain » Thu, May 21, 2020, 5:58 PM UTC

hnrodey wrote: post #60

I'm very well versed on RabbitMQ. We use it internally in a .NET codebase. Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some late…

Can you talk a little bit about how you've managed your RabbitMQ infrastructure? Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit?

I'm looking to pitch adding a message queue to our infrastructure (at a .Net shop on Azure), and I'm sure there will be some questions about the comparisons between the two. Unfortunately, that's been tough to really track down.

Re: An introduction to RabbitMQ

#112

Post by wenc » Thu, May 21, 2020, 6:01 PM UTC

tombert wrote: post #90

I feel like RabbitMQ is sort of the "swiss army knife" of message queues, and I mean that in the nicest way possible. People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across. I recently did a proj…

Rabbit doesn't have Kafka's ability to massively distribute and scale (it does have a distributed story but from what I hear few explore it). But Rabbit also supports more complex use cases than Kafka because its messaging protocol (AMQP) is more intelligent. Unless you're a "web-scale"/s company, Rabbit's scale even on one node is likely enough.

I've been using Rabbit in production for RPC and pub/sub for the past 5 years (single instance running on a non-dedicated VM, medium traffic) and its been pretty easy to setup and has been pretty reliable in practice.

I've always been concerned about losing messages, and I did have to learn to turn on persistence and durability for messages to survive server interruptions, but it was easy enough. Message acknowledgements are also a nice feature, and Rabbit is able to achieve at-least-once messaging semantics.

Re: An introduction to RabbitMQ

#113

Post by zkirill » Thu, May 21, 2020, 6:05 PM UTC

Would anyone using RabbitMQ as a replacement for GCM/FCM on Android mind sharing their experiences?

Re: An introduction to RabbitMQ

#114

Post by danjc » Thu, May 21, 2020, 6:10 PM UTC

hnrodey wrote: post #60

I'm very well versed on RabbitMQ. We use it internally in a .NET codebase. Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some late…

This. We used Rabbit in our platform for a few years and it was an absolute disaster. Network partitions mainly. In retrospect I'm sure we were doing it wrong but that really wasn't obvious at the time.

Re: An introduction to RabbitMQ

#115

Post by wallstprog » Thu, May 21, 2020, 6:15 PM UTC

dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

For now, I'll just address the one point -- obsolete? NOT!

We've been working with ZeroMQ a lot over the past couple of years, and have gotten to know some of the maintainers -- we've been very favorably impressed by their ability and dedication.

Pieter Hintjens was the "voice" of ZeroMQ, and with his passing things have gotten a bit quieter, but no less active. (Just take a look at the commit log: https://github.com/zeromq/libzmq/commits/master).

Re: An introduction to RabbitMQ

#116

Post by zerr » Thu, May 21, 2020, 6:17 PM UTC

crad wrote: post #38

Using the opportunity to pimp my book, RabbitMQ in Depth: https://www.manning.com/books/rabbitmq-in-depth :)

This is HN - we need a sales chart! :)

Re: An introduction to RabbitMQ

#117

Post by webscalist » Thu, May 21, 2020, 6:31 PM UTC

RabbitMQ has huge learning curve if you're trying to build a worker queue.

First, you'll learn about ack/noack and get the worker ack on success.

Then, you'll learn about dead letter queue ... etc for delayed retries.

Now, you'll have a topic exchange and a bit hairy routing in place using wildcards.

And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and actual worker queue ... ).

Then you rewrite your service in python and use Celery or something.

It's nearly impossible to get RabbitMQ working correctly within few months.

And I forgot about HA. Paying for hosted RabbitMQ might be better. But CloudAMQP in particular could be tricky as well. It can run out of AWS IOPS and your production gets hosed.

Also setting up monitoring on queue health, shoveling error queues ... etc take time to learn and apply. Be careful about routing keys when you shovel error queue to a topic.

Re: An introduction to RabbitMQ

#118

Post by heinrichhartman » Thu, May 21, 2020, 6:46 PM UTC

lulf wrote: post #81

heinrichhartman wrote: post #44

Earlier quoted context omitted.

You might find it interesting to note, that Peter Hintjens, was one of the core authors of the AMQP 0-9-1 Specification [1], that RabbitMQ is implementing. ZeroMQ was born out of a frustration with complex routing patterns and the need for a broker-less architecture for maximal performance message delivery. [1] https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf

I'll add that the AMQP 1.0 spec (supported in Rabbit using a plugin) is a peer-to-peer protocol that supports both the traditional broker use case, 'direct' p2p messaging and opens some interesting uses of message routers like Apache Qpid Dispatch Router.

I am no export but I have heard PH say, that it's much worse than the AMQP-0.9 Spec. It's a design-by-comitte thing, where he was sidelined.

Re: An introduction to RabbitMQ

#119

Post by spamizbad » Thu, May 21, 2020, 6:47 PM UTC

danielstocks wrote: post #37

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

You're usually stuck polling for stuff like that. I'm a big fan of using things like Advanced Python Scheduler for those sorts of tasks: https://apscheduler.readthedocs.io/en/stable/

Re: An introduction to RabbitMQ

#120

Post by atombender » Thu, May 21, 2020, 6:49 PM UTC

airfreak wrote: post #74

atombender wrote: post #41

Earlier quoted context omitted.

Adding to what the sibling comment say, be careful about buying into RabbitMQ's clustering; having run it for years, I found it to be extremely brittle. We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but t…

The old network partition problems people remember about RabbitMQ are solved by quorum queues.

Yes, but quorum queues don't have many of the features of classic mirrored queues.