Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

141–150 of 263 posts

Re: An introduction to RabbitMQ

#141

Post by [deleted] »

ekimekim wrote: post #85
dkersten wrote: post #72

Earlier quoted context omitted.

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from. The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it. I de…

[deleted]

Re: An introduction to RabbitMQ

#142

Post by sciurus »

jackvanlightly wrote: post #88
ekimekim wrote: post #85

Earlier quoted context omitted.

This came up in several other threads here: Don't use RabbitMQ's clustering. It's surprisingly brittle and hard to recover from. The accepted wisdom that I've seen is to run a single broker with a completely independent hot spare. But of course switching over to your hot spare will violate most of the guarentees that Rabbit gives you around durability, ordering etc, so you have to be very careful how you use it. I de…

Rabbit dev here. We released quorum queues a few months ago. It's a Raft based replicated queue that addresses all the old problems. https://www.rabbitmq.com/blog/2020/04/20/rabbitmq-gets-an-ha...

Thanks for all your work on RabbitMQ, and for your great blog posts about it and other messaging systems.

For anyone who wants to understand the potential complexities of HA RabbitMQ, spend some time reading https://jack-vanlightly.com/blog/2018/8/31/rabbitmq-vs-kafka...

Re: An introduction to RabbitMQ

#143

Post by symplee »

webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

Can anyone recommend an easier alternative?

Re: An introduction to RabbitMQ

#144

Post by vasco »

webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

Celery can be backed by RabbitMQ, not sure if that's what you meant, but all of what you described can be abstracted away. I didn't have the same experiences with months taken to get up to speed. Moreover, at work RabbitMQ is probably our most stable underlying tool, perhaps toe to toe with Redis. And that's saying a lot, since I consider Redis to almost be a piece of art in how great of a tool it is.

Back to RabbitMQ though, we run a HA 2 node deployment (just one active writer) and have been for over 3 years, requiring minimal changes or any kind of maintenance whatsoever, has scaled to hundred plus queues, going from some with super high numbers of messages per second, some with only tens of messages per day. Some queues stay low and process fast, others are heavy jobs that get enqueued all at once and generate hundreds of thousands of jobs.

Sure, if you have a service that interacts with disks you should have automated a monitor that cover your IOPS consumption, but I don't see how that's specific to RabbitMQ, you should be doing this for all your instances.

All in all, these are two identic instances, one active, one failover, and in a world of Kafkas and Pulsars and understanding the ins and outs of SQS pricing and capacity allocation, RabbitMQ is a tool that I consider simple to administer and allows me to sleep at night.

Interesting how the same tool can evoke such different reactions, but whatever works - works.

Re: An introduction to RabbitMQ

#145

Post by mcsoft »

symplee wrote: post #143
webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

Can anyone recommend an easier alternative?

http://nats.io

Re: An introduction to RabbitMQ

#146

Post by danthemanvsqz »

danielstocks wrote: post #37

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

You schedule tasks very easily with the Celery Beat Scheduler. I've used it in production system to kick off big jobs and notifications.

https://docs.celeryproject.org/en/v2.3.3/userguide/periodic-...

Re: An introduction to RabbitMQ

#147

Post by polygotdomain »

hnrodey wrote: post #127
polygotdomain wrote: post #111

Earlier quoted context omitted.

Can you talk a little bit about how you've managed your RabbitMQ infrastructure? Also if you've done any comparison to Azure Message Queues and what were the pros and cons against Rabbit? I'm looking to pitch adding a message queue to our infrastructure (at a .Net shop on Azure), and I'm sure there will be some questions about the comparisons between the two. Unfortunately, that's been tough to really track down.

>Can you talk a little bit about how you've managed your RabbitMQ infrastructure? From a ten thousand foot view, two or three node clusters running in non-prod environments on virtual machines running Windows. In Prod, three node clusters on Windows virtual machines. All work to install and configure RabbitMQ is done manually. Sadly enough. I'm on the application/architecture side of this equation but I know enough a…

Thanks for the response. From the other responses in this thread, it seems like the admin of the nodes/cluster is not overly onerous. Would you agree with that statement? Also, being a .Net shop, the Windows VMs make sense, but is there any tradeoffs to running Rabbit on Windows, as opposed to Linux?

I think part of the sell is how we would manage the admin component of a Message Queue, which tilts things towards Azure Message Queues as it's PaaS. We're mostly IaaS at the moment, and starting to see some of the admin overhead that comes with managing that infrastructure ourselves. We're not ready to jump onto a PaaS solution for the things we've grown accustomed to managing, but for something brand new, I think my company would be open to it.

Architecturally, we'd lean on it initially for background job processing, which is currently at a scale where our homegrown, db-backed solution is starting to show it's weaknesses. Once it's in place though, I think it could leveraged as a key component to decouple subsections our application and give us more flexibility with scaling and deployment.

Re: An introduction to RabbitMQ

#148

Post by csours »

webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

I'm curious if you are comparing this to a non-queue solution or to a different queue system?

Re: An introduction to RabbitMQ

#149

Post by ramchip »

shoo wrote: post #135
atombender wrote: post #41

Earlier quoted context omitted.

Adding to what the sibling comment say, be careful about buying into RabbitMQ's clustering; having run it for years, I found it to be extremely brittle. We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but t…

> lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was I can offer a similar anecdote: we started seeing rabbitmq reporting alleged cluster partitions in production after enabling TLS between rabbitmq nodes, where manual recovery was needed each time. After a bit…

Erlang/OTP-22 (released last year) introduced TLS distribution optimizations and message fragmentation which sound very related to the problem you saw:

http://blog.erlang.org/OTP-22-Highlights/

The fragmentation in particular addresses the problem where a large message would block all other messages, including heartbeats, and cause nodes to look “down” when they’re not.

Re: An introduction to RabbitMQ

#150

Post by Turbots »

webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

Use a library... Spring Cloud Stream for example lets you connect to topics, create consumer groups, have automatic dead letter queues, etc etc... Why would you use RabbitMQ in such a manual way? Even in Python there's the Pika library for easy consumption of Rabbit
Post reply on HN