Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

41–50 of 263 posts

Re: An introduction to RabbitMQ

#41

Post by atombender »

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

Adding to what the sibling comment say, be careful about buying into RabbitMQ's clustering; having run it for years, I found it to be extremely brittle.

We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but they don't solve the core problem; the only way to run mirrored queues ("classic mirrored queues", as they're not called) reliably is to disable automatic recovery, and then you have to manually repair RabbitMQ every time this happens. If you care about integrity, you can use the new quorum queues instead, which use a Raft-based consensus system, but they lack a lot of the features of the "classic" queues. No message priorities, for example.

I've never used federation or Shovel, which are different features with other pros/cons.

If you're willing to lose the occasional message under very high load, NATS [3] is absolutely fantastic, and extremely fast and easy to cluster. Alternatively, NATS Streaming [4] and Liftbridge [5] are two message brokers built on top of NATS that implement reliable delivery. I've not used them, but heard good things.

[1] https://www.rabbitmq.com/partitions.html

[2] https://www.rabbitmq.com/quorum-queues.html

[3] https://nats.io/

[4] https://docs.nats.io/nats-streaming-concepts/intro

[5] https://github.com/liftbridge-io/liftbridge

Re: An introduction to RabbitMQ

#42

Post by hhhlgp »

dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

JeroMQ has been nothing but a pleasure. I don't see ZeroMQ being obsolete in its forked forms anytime soon.

Re: An introduction to RabbitMQ

#44

Post by heinrichhartman »

dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

You might find it interesting to note, that Peter Hintjens, was one of the core authors of the AMQP 0-9-1 Specification [1], that RabbitMQ is implementing.

ZeroMQ was born out of a frustration with complex routing patterns and the need for a broker-less architecture for maximal performance message delivery.

[1] https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf

Re: An introduction to RabbitMQ

#45

Post by atombender »

dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

The "MQ" in "ZeroMQ" is misleading, so this is an apples-to-oranges comparison. ZeroMQ is a socket abstraction that allows you to build apps that send messages to each other. RabbitMQ is a reliable message queue broker; a central server that stores messages and that clients connect to in order to push/pop them.

Re: An introduction to RabbitMQ

#46

Post by olikas »

aeyes wrote: post #36
olikas wrote: post #18

Earlier quoted context omitted.

Author here. I've seen quite a lot messages going through RabbitMQ. I wouldn't worry too much about scaling, because the possibilities depend very much on the architecture. With some tuning RabbitMQ can take you a long way. I would give clustering a go and see where the limits are before exploring more complicated architectures like federation.

Could you explain how RabbitMQ clustering is going to improve performance? For how it works I would expect it to lower performance.

With clustering, you can have more nodes and you can shard (distribute) your queues over the cluster. You don't need to mirror every queue on every node. But you are right, mirroring alone will add more load.

Re: An introduction to RabbitMQ

#47

Post by jalons »

dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

No. The only thing ZeroMQ and RabbitMQ have in common are the letters M and Q.

RabbitMQ is a messaging system. ZeroMQ is sockets on steroids.

Re: An introduction to RabbitMQ

#48

Post by aeyes »

vel0city wrote: post #29
halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

I've been running RabbitMQ on pretty small VMs for a long time. RabbitMQ doesn't need a lot of resources per message, even with very small VMs (512MB RAM, single CPU) I've seen it handle peaks of many thousands of messages a second without running into problems. Give it a bit of beefy hardware and it'll probably handle whatever load you were thinking, unless you're saturating 10gig links with messages or something. R…

CPU usage highly depends on the number of connected clients, not that much on message throughput. You can experiment with the excellent rabbitmq-perf-test tool to get some ballpark numbers.

I have a system that only pushes 5k messages per second but it needs 32 cores.

Re: An introduction to RabbitMQ

#49

Post by [deleted] »

jialutu wrote: post #17
speedgoose wrote: post #9

Earlier quoted context omitted.

I do and it's great. It doesn't have some MQTT features sush as persisted messages or QOS 2 but if you don't need that, it's a fine MQTT broker.

Wait, what are you talking about? RabbitMQ does have persistent messages, you just need to set the queue as "durable", and the messages persist even during failures.

[deleted]

Re: An introduction to RabbitMQ

#50

Post by zonk_ »

dkersten wrote: post #39
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ.

I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retries, but there are some "RabbitMQ patterns" to make it work. For a simple message broker/queue system, it's great and the docs are also great.

Post reply on HN