An introduction to RabbitMQ • Hacker News Board

Re: An introduction to RabbitMQ

#121

Post by ecoqba11 » Thu, May 21, 2020, 6:55 PM UTC

dkersten wrote: post #39

akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

Been using RabbitMQ for a lot of projects in production. It can handle quite a lot data and this thing never fails. Sometimes it can be running for an entire year and we force restart just because.

Re: An introduction to RabbitMQ

#122

Post by rawoke083600 » Thu, May 21, 2020, 6:57 PM UTC

Man I love this piece of software.. We used it as a bare-bones msg queue FIFO and some FAN-OUT patterns. Basically only scratch the service of what is possible. But this beast ran our ETL distributed update system at PriceCheck (S.A largest price comparison service). Haven't worked there now for a few years but back then the RabbitMQ was rock-solid for us !

Re: An introduction to RabbitMQ

#123

Post by davidcorbin » Thu, May 21, 2020, 6:58 PM UTC

RabbitMQ has been awesome in my experience. One of the few tools that just works and has a super useful management web interface and Prometheus support among other plugins.

For those noting HA and scalability, it not meant for those use cases where (virtually infinite) horizontal scalability are the biggest concern. If you need horizontal scalability at a massive scale, use Kafka. But for the majority of cases, you can get away with limited scalability and the prod setup, development experience, and reliability of RabbitMQ are unmatched from my experience.

Re: An introduction to RabbitMQ

#124

Post by rawoke083600 » Thu, May 21, 2020, 6:59 PM UTC

ecoqba11 wrote: post #121

dkersten wrote: post #39

Earlier quoted context omitted.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

Been using RabbitMQ for a lot of projects in production. It can handle quite a lot data and this thing never fails. Sometimes it can be running for an entire year and we force restart just because.

Yup been my production experience as well ! Super solid system !

Re: An introduction to RabbitMQ

#125

Post by dkersten » Thu, May 21, 2020, 7:01 PM UTC

heipei wrote: post #99

dkersten wrote: post #39

Earlier quoted context omitted.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

My go-to solution for fault-tolerant message queues is nsq ( https://nsq.io/ ). nsq works differently from most other message queues in that it's supposed to be run in a distributed fashion, i.e. one nsqd running wherever messages are produced. That way you have a lightweight and fast local message queue that you can push messages to and not worry about network connectivity. You can use nsqlookupd to find the distrib…

Thanks for the recommendation! That looks pretty nice and “ops friendly” is definitely a plus. I will investigate this further.

Re: An introduction to RabbitMQ

#126

Post by rawoke083600 » Thu, May 21, 2020, 7:04 PM UTC

rhodin wrote: post #71

harel wrote: post #2

RabbitMQ is one of those pieces of software I usually forget are there. I can't remember having to deal with any rabbit issue in last few years.

This might be the Achilles heel of RabbitMQ. It works so well that people forget it for years, and then they have forgotten how to upgrade it, etc. :)

Lol this ! WRITE down that rabbitmq-web-admin passwd. After the setup and first few weeks of checking the speed of your queues you will forget about it and try to login in 1 year later :)

Re: An introduction to RabbitMQ

#127

Post by hnrodey » Thu, May 21, 2020, 7:06 PM UTC

polygotdomain wrote: post #111

hnrodey wrote: post #60

I'm very well versed on RabbitMQ. We use it internally in a .NET codebase. Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some late…

Can you talk a little bit about how you've managed your RabbitMQ infrastructure? Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit? I'm looking to pitch adding a message queue to our infrastructure (at a .Net shop on Azure), and I'm sure there will be some questions about the comparisons between the two. Unfortunately, that's been tough to really track down.

>Can you talk a little bit about how you've managed your RabbitMQ infrastructure?

From a ten thousand foot view, two or three node clusters running in non-prod environments on virtual machines running Windows. In Prod, three node clusters on Windows virtual machines.

All work to install and configure RabbitMQ is done manually. Sadly enough.

I'm on the application/architecture side of this equation but I know enough about our infrastructure to perhaps answer follow-ups or more specific questions.

Our application is single tenant (so each customer is deployed in their own isolated area) so we use virtual hosts to isolate each customer within the cluster.

>Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit?

Definitely looked in to the Azure native queueing options but it's been awhile. Azure Message Queues is an AMQP compliant messaging system that seems fairly robust. To be transparent, I have no production experience with this product. If your company/department is in to managing virtual machines then they might want/prefer to go with RabbitMQ. However, if they're in to PaaS systems then I'd probably roll with Azure Message Queues and never look back.

Re: An introduction to RabbitMQ

#128

Post by tombert » Thu, May 21, 2020, 7:09 PM UTC

wenc wrote: post #112

tombert wrote: post #90

I feel like RabbitMQ is sort of the "swiss army knife" of message queues, and I mean that in the nicest way possible. People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across. I recently did a proj…

Rabbit doesn't have Kafka's ability to massively distribute and scale (it does have a distributed story but from what I hear few explore it). But Rabbit also supports more complex use cases than Kafka because its messaging protocol (AMQP) is more intelligent. Unless you're a "web-scale"/s company, Rabbit's scale even on one node is likely enough. I've been using Rabbit in production for RPC and pub/sub for the past 5…

Yeah, I don't dispute that for certain usecases, Kafka is definitely the better choice, use the right tool for the right job.

That said, for most small to medium-large tasks, Rabbit will handle things without much trouble, making it a good fit for most common usecases.

Re: An introduction to RabbitMQ

#129

Post by niffydroid » Thu, May 21, 2020, 7:12 PM UTC

We use AWS SQS and Rabbit. At our scale, SQS is easy peasy and we can wrapper it to make http calls instead of using SQS, as we're using AWS Beanstalk workers. SQS is generally quicker to get up and running with and we can have metrics out the box. With rabbit we use it for some other stuff and it works just fine, it's when things go into a black hole we struggle, but that's our lack of knowledge.

Depending on your scale, we find SQS is cheaper than a managed rabbit service. Although I'd be interested in using kafka!

Re: An introduction to RabbitMQ

#130

Post by t_sawyer » Thu, May 21, 2020, 7:12 PM UTC

My only experience with RabbitMQ is managing an Openstack environment. In that environment, it's a huge resource hog and we had to put it on 3 separate bare metal instances to keep it stable.