Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

101–110 of 263 posts

Re: An introduction to RabbitMQ

#101

Post by robbyt »

wegs wrote: post #65

My general problem is that it's really hard to figure out which architecture is right for which system. There's a different architecture for: * one queue with billions of messages * a millions of queues with small numbers of messages per queue * many queues with many messages per queue There are also different topologies: * Anyone can send a message to anyone (O(n^2) queues) * One publisher with millions of subscribe…

All software breaks at some point. If you're dealing with this scale of load, it's mandatory to perform synthetic load testing to validate, otherwise you're just guessing what the breakage threshold will be.

Re: An introduction to RabbitMQ

#102

Post by Dowwie »

I've got a connection/channel question for those who have built solutions with rabbitmq-- how did you decide as to how many connections and channels-per-connection to use? Does connection pooling even make sense for RabbitMQ? My impression is that channel pooling may make more sense. Thoughts?

Re: An introduction to RabbitMQ

#103

Post by pbhowmic »

dkersten wrote: post #39
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

RabbitMQ, even a single-node RabbitMQ, has a hard time going down. You are more likely to have your server/container go down long before RabbitMQ node goes down. That being said, if you want to have a clustered solution with nodes being in different DCs, configure shoveling (https://www.rabbitmq.com/shovel.html) or for a simpler solution, use a private VPN to interconnect the RabbitMQ nodes. I would go for the latter.

Re: An introduction to RabbitMQ

#104

Post by dirtydroog »

trevyn wrote: post #52
dirtydroog wrote: post #26

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

I think they’re slightly different solutions — ZeroMQ works without a broker, RabbitMQ requires a server process. If you use the brokerless model, there was a bit of drama over ZeroMQ — the original technical developer (Martin Sustrik) left and created a successor, nanomsg, with what he learned. At some point, Martin lost interest, and Garrett D’Amore took over maintenance and did a rewrite called nng. Both the old n…

Yes, my handwavy reading of the situation was that he left due to issues with zeromq that he couldn't/wasn't allowed 'fix'. Then Peter Hintjens unfortunately died a few years back. I haven't heard about nng, so thanks for that, I'll check it out.

ZeroMQ certainly isn't perfect, for example there's no way to tell if a message was successfully written to a PUB socket, or if it was dropped (just one minor issue)

https://stackoverflow.com/questions/13891682/detect-dropped-...

Anyway, This is digressing from the main topic

Re: An introduction to RabbitMQ

#105

Post by fernandotakai »

tombert wrote: post #90

I feel like RabbitMQ is sort of the "swiss army knife" of message queues, and I mean that in the nicest way possible. People will compare it to Kafka, claiming that its pubsub is faster than Rabbit's, but that's sort of missing the point: Rabbit thrives because it's easy to set up, will work well for 99% of cases, and handles nearly every kind of distributed problem you're likely to come across. I recently did a proj…

it's also super duper stable even on default configs. it's one of my favorite softwares ever.

Re: An introduction to RabbitMQ

#106

Post by Plugawy »

Dowwie wrote: post #102

I've got a connection/channel question for those who have built solutions with rabbitmq-- how did you decide as to how many connections and channels-per-connection to use? Does connection pooling even make sense for RabbitMQ? My impression is that channel pooling may make more sense. Thoughts?

An application usually has one connection, and many channels. Our pattern is to dedicate one channel for all publishing and then N channels mapped to consumer threads.

You don't have to pool connections as channels are multiplexed by them.

Things to watch out for:

- opening too many channels - these map to Erlang processes and can overwhelm your server if you go over ulimits - sharing consumer channels between threads - you might see weird behavior (e.g. acking wrong messages etc)

We've built own library/framework for creating resilient consumers, and it enforces mapping 1:1 channels and consumer threads, as well as automatic reconnections and channel clean ups.

Re: An introduction to RabbitMQ

#107

Post by fernandotakai »

atombender wrote: post #41
halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

Adding to what the sibling comment say, be careful about buying into RabbitMQ's clustering; having run it for years, I found it to be extremely brittle. We often lost entire queues because a small network blip caused RabbitMQ to think there was a network partition, and when the other nodes became visible, RabbitMQ has no reliable way to restore its state to what it was. It has a bunch of hacks to mitigate this, but t…

it used to be really bad, that's super true.

nowadays? it's actually quite simple to setup and works pretty well (source: i know two different companies that setup clustering recently and both had good experiences with no downtime).

Re: An introduction to RabbitMQ

#108

Post by dkersten »

zonk_ wrote: post #86
dkersten wrote: post #72

Earlier quoted context omitted.

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

> Was it easy to setup in terms of reliability and failover? To be honest, the current setup was set up by colleagues who have even less experience than me and it's still running flawlessly. Iirc, it's just two instances that are behind a load balancer and the consumer just consumes from both, but I'm not super certain on that. I've tested the cluster functionality to see how to set it up and it worked fine for me, b…

Thanks for the detailed response (and to everyone else who responded too!), I appreciate it! I will prototype something and play around and see how it handles different situations when I get time. I've also just bought the book mentioned elsewhere, so hopefully I can get up to speed quickly. It does sound that my original impression about it being complex to run/maintain was perhaps overblown. That's good, because from a features point of view, RabbitMQ seemed like a good fit for the things I want an MQ solution for.

Re: An introduction to RabbitMQ

#109

Post by MR4D »

pjc50 wrote: post #83
MR4D wrote: post #73

Earlier quoted context omitted.

> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally

It always was a message queue in a very literal sense. There's a lot of work in mailer-daemons to ensure that email has as reliable as possible delivery in a store-and-forward system..

You're correct - I left out the word "universal" accidentally, which would have made my intent much more clear.

Thanks for catching that.

Re: An introduction to RabbitMQ

#110

Post by carterklein13 »

nailer wrote: post #91
carterklein13 wrote: post #34

Earlier quoted context omitted.

When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? At least from my experience, HTTP microservices are still very common, especially when using things like AWS Lambdas. I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just us…

> When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? They could have stayed trying to do continually running microservices on HTTP. > I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc I do too. > more and more I see lightweight systems going completely serverless and just using lambdas - which are HTT…

To be fair, I started working post-2015, so I've actually never come face-to-face with a long running HTTP microservice backend... what would something like that even look like? I'm thinking of systems I've worked on that use a messaging queue, but that only rely on HTTP requests - is that what it would be? So like, I'd make a request to a microservice behind an endpoint, which in turn would make requests to 3 more microservices behind other endpoints? If so, I'm certainly glad that idea isn't cool anymore because that seems greatly inefficient :)
Post reply on HN