Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

81–90 of 263 posts

Re: An introduction to RabbitMQ

#81

Post by lulf »

heinrichhartman wrote: post #44
dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

You might find it interesting to note, that Peter Hintjens, was one of the core authors of the AMQP 0-9-1 Specification [1], that RabbitMQ is implementing. ZeroMQ was born out of a frustration with complex routing patterns and the need for a broker-less architecture for maximal performance message delivery. [1] https://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf

I'll add that the AMQP 1.0 spec (supported in Rabbit using a plugin) is a peer-to-peer protocol that supports both the traditional broker use case, 'direct' p2p messaging and opens some interesting uses of message routers like Apache Qpid Dispatch Router.

Re: An introduction to RabbitMQ

#83

Post by pjc50 »

MR4D wrote: post #73
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally

It always was a message queue in a very literal sense.

There's a lot of work in mailer-daemons to ensure that email has as reliable as possible delivery in a store-and-forward system..

Re: An introduction to RabbitMQ

#84

Post by airfreak »

jugg1es wrote: post #80

I love RabbitMQ but deploying/managing a cluster can be tricky. We had problems with network partitioning and since we didn't really need a cluster for performance reasons - only availability - we switched to a single node.

Try the new quorum queues, they don't have those issues.

Re: An introduction to RabbitMQ

#85

Post by ekimekim »

dkersten wrote: post #72
zonk_ wrote: post #50

Earlier quoted context omitted.

We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ. I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retri…

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from.

The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it.

I desperately want to like Rabbit (and have used it heavily in the past) but right now I wouldn't use it if I can get away with anything else, it just has no real HA story.

Re: An introduction to RabbitMQ

#86

Post by zonk_ »

dkersten wrote: post #72
zonk_ wrote: post #50

Earlier quoted context omitted.

We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ. I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retri…

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

> Was it easy to setup in terms of reliability and failover?

To be honest, the current setup was set up by colleagues who have even less experience than me and it's still running flawlessly. Iirc, it's just two instances that are behind a load balancer and the consumer just consumes from both, but I'm not super certain on that.

I've tested the cluster functionality to see how to set it up and it worked fine for me, but I have no experience with that in production, but other people in this thread don't seem to be too happy with it, so ymmv.

> Is there anything I should keep in mind for running it in production?

Nothing special that you wouldn't do otherwise, when getting to know a new component/microservice. Just check out the get started section[1] on their page in the appropriate language and play around with a small setup. Get familiar with the libraries to connect and send/queue/fetch stuff and the topics. Make sure to use your brainpower before you set it up to handle all eventualities and is set up exactly how you want it to act (ACK/NACK, what happens if a sender/consumer dies, etc.), because then you set it up once and probably never touch it again.

One thing I'm not really sure on and what I haven't really answered myself yet: there might be some "logic" to your RabbitMQ instance, depending on which metadata you add to each message (e.g. retries). If you have such logic, it might be better to have a service around the RabbitMQ instance, otherwise this logic ends up in your code base of you actual solution and that might not be wanted and maybe harder to maintain. But I'm not so sure myself on that one.

Oh yeah, and check out some patterns for your needs. There are for example multiple ways to implement retries, for example with a queue for queues, etc. But if you API is REST based, everything should be straightforward.

[1]: https://www.rabbitmq.com/getstarted.html

Re: An introduction to RabbitMQ

#87

Post by ketralnis »

dkersten wrote: post #39
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

We use it for more or less everything at reddit. Almost every user action corresponds to a rabbit queue

Re: An introduction to RabbitMQ

#88

Post by jackvanlightly »

ekimekim wrote: post #85
dkersten wrote: post #72

Earlier quoted context omitted.

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from. The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it. I de…

Rabbit dev here. We released quorum queues a few months ago. It's a Raft based replicated queue that addresses all the old problems. https://www.rabbitmq.com/blog/2020/04/20/rabbitmq-gets-an-ha...

Re: An introduction to RabbitMQ

#89

Post by guptaneil »

MR4D wrote: post #73
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally

I think this is what a lot of people who complain about Slack don't get. It's just a better message queue for your business. The fact that you can funnel all your business events, regardless of whether they originate from humans or bots, into one place and then each worker (again, either human or bot) can subscribe/filter/react to relevant events is super powerful. However, if you try to use it as a corporate SMS platform or email replacement, you will very quickly feel overwhelmed because both of those message queues are designed for much lower throughput.

Re: An introduction to RabbitMQ

#90

Post by tombert »

I feel like RabbitMQ is sort of the "swiss army knife" of message queues, and I mean that in the nicest way possible.

People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across.

I recently did a project with Rabbit on my home server, and while the project had some issues, the issues were never Rabbit.

Post reply on HN