Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

241–250 of 263 posts

Re: An introduction to RabbitMQ

#241

Post by tracker1 »

olikas wrote: post #151
polygotdomain wrote: post #147

Earlier quoted context omitted.

Thanks for the response. From the other responses in this thread, it seems like the admin of the nodes/cluster is not overly onerous. Would you agree with that statement? Also, being a .Net shop, the Windows VMs make sense, but is there any tradeoffs to running Rabbit on Windows, as opposed to Linux? I think part of the sell is how we would manage the admin component of a Message Queue, which tilts things towards Azu…

RabbitMQ runs perfectly fine on Windows too. As others mentioned in the comments, RabbitMQ supports a great variety of use cases. If you want to reach out for help, you can find my contact in the article.

Download and set aside a copy of the erlang and rabbitmq installers if you're running on windows... I've had issues on many occassions with the erlang installer being unavailable or very slow to download.

Re: An introduction to RabbitMQ

#242

Post by smoe »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

Sounds to me like the problem is, that the company taking out the cannon to shoot at sparrows and that nobody seem to have bothered to provide working dev environments?

It feels common that people think they their every problem requires a planetary scale solution or one that handles every conceivable case that could occur before the heat death of the universe. Be that because your a startup and think you are going to need to support a hundred million concurrent users on launch day, or because your an enterprise and think because your are oh so important you need to use the same tools other important companies use.

My own anecdote: When we refactored our early stage app we had a huge mess with the Redis based queue system. It was one of the biggest sources of errors and a massive pain to troubleshoot or even to monitor what is going on. So we investigated in a bunch of different solutions including all the usual contenders, and we ended up with: Let's just ditch the messages / queues altogether for now and just do boring old cron like jobs invoking internal api endpoints in regular intervals. This made 9 out of 10 cases a lot easier to maintain, and for the remainder, while temporary more difficult, we introduced a queue again at a much later stage.

I'm not saying you shouldn't use and you can't benefit of RabbitMQ at small scales or any technology for that matter. But I think too often in tech decision making ones own or the companies perceived importance, what is cool or what would look good on a CV takes precedence over what really fits the problem in context.

Re: An introduction to RabbitMQ

#243

Post by lllr_finger »

snapetom wrote: post #79
halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

I've used both. I was introduced to Rabbit at one job and at another, was "fed" Kafka during a selection process. At that time, I was definitely not opposed to Kafka because, hey new resume item. I ended up yearning for Rabbit for three reasons. 1) Much easier to implement and maintain for small to medium architectures. However, war stories I've heard is that it starts to become a hassle for large clustering architec…

Kafka is one of those things where if you're new to it, especially if you're coming from Rabbit or similar, you might tend to assume the happy path - exactly once delivery. This is a bad mistake (whether that's possible and to what definition is not a debate I'd like to dive into now). What you should expect from Kafka is at least once delivery.

There will be times when you lose offsets or when you actually want to replay every message, so take an hour and figure out what that means to your app. It's usually only a few lines of code in your consumer that compares source timestamps, but it's by far the most beneficial thing you can do when working with Kafka in my experience.

It's also relatively easy to hit "tens of thousands" messages/second, especially in replay or bootstrapping scenarios, and that's when Kafka becomes useful to the non-FAANG companies.

Re: An introduction to RabbitMQ

#245

Post by kazinator »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

ZeroMQ compiles to well over 600 kilobytes of machine code (just on 32 bit x86).

  $ size /usr/lib/i386-linux-gnu/libzmq.so
     text    data     bss     dec     hex filename
   662134   12952      24  675110   a4d26 /usr/lib/i386-linux-gnu/libzmq.so
Yikes, is there an operating system kernel with virtual memory, USB support, a few filesystems and a TCP/IP stack hiding in there?

It's like 35% of glibc:

  $ size /lib/i386-linux-gnu/libc.so.6 
     text    data     bss     dec     hex filename
  1917549   11624   11112 1940285  1d9b3d /lib/i386-linux-gnu/libc.so.6
Which is itself heavily bloated, trying to provide as much POSIX as it can.

Re: An introduction to RabbitMQ

#246

Post by hnrodey »

fmorel wrote: post #192
hnrodey wrote: post #60

I'm very well versed on RabbitMQ. We use it internally in a .NET codebase. Anyone considering RabbitMQ needs to read up on "network partitions", how to build your cluster to avoid them (odd number of nodes and pause_minority), your recovery strategy for when a network partition occurs (it will occur), your personal/organizational tolerance for message loss and a plan for how you will upgrade your cluster at some late…

If you're using RabbitMQ on .Net, I highly recommend using NserviceBus. It's made working with queues so easy . It handles maintaining a connection and retrying/acknowledging messages for you.

Hindsight is the best site. That's definitely what I would do if I was starting a new project using RabbitMQ. Although I'll defend myself on this front; I inherited our RabbitMQ project from the developer who left the company 7/8 of the way through the implementation. I had the "make it work" directive and not the decision making luxury he had from the beginning.

Re: An introduction to RabbitMQ

#247

Post by shaklee3 »

jonathanoliver wrote: post #209
carterklein13 wrote: post #40

Would anyone be able to explain the benefits of RabbitMQ over NATS? As far as I've seen, it's really just that RabbitMQ is more feature-rich, which I personally feel like isn't that crucial, as frankly many systems are not going to take advantage of those more complex functionalities anyway.

Durability. If you need to push messages that don't get lost, RabbitMQ is a pretty solid choice. In years past the clustering situation wasn't great and there was some potential for message lost and that seems to be resolved now with quorum queues, but the biggest different between NATS and RMQ is the durability guarantees and the at-least-once delivery guarantees that RMQ has. NATS is more like ZeroMQ in that it exp…

This isn't true anymore. Nats streaming has persistence, so the OP's question still remains

Re: An introduction to RabbitMQ

#248

Post by shaklee3 »

jonathanoliver wrote: post #199
ies7 wrote: post #171

Earlier quoted context omitted.

I heard a lot of praise about Nats, but isn't it more like a kafka alternative? Someone new need to spend sometime grasping the stream concept.

NATS by itself is designed to be more of an always-on style queuing system (the term they use is "dial tone") but doesn't handle node failures by itself. If you're looking for a Kafka-flavored NATS, there's a new release I saw recently called LiftBridge that adds some durability to the NATS protocol.

Someone mentioned this already below, but nats streaming also adds durability.

Re: An introduction to RabbitMQ

#249

Post by jonathanoliver »

shaklee3 wrote: post #247
jonathanoliver wrote: post #209

Earlier quoted context omitted.

Durability. If you need to push messages that don't get lost, RabbitMQ is a pretty solid choice. In years past the clustering situation wasn't great and there was some potential for message lost and that seems to be resolved now with quorum queues, but the biggest different between NATS and RMQ is the durability guarantees and the at-least-once delivery guarantees that RMQ has. NATS is more like ZeroMQ in that it exp…

This isn't true anymore. Nats streaming has persistence, so the OP's question still remains

My understanding is that NATS (a protocol) and NATS streaming were related but separate:

https://github.com/nats-io/nats-site/issues/217

(The issue is from 2017 but illustrates a distinction)

Re: An introduction to RabbitMQ

#250

Post by lukebakken »

I highly recommend all of Jack's blog posts about RabbitMQ - https://jack-vanlightly.com/blog/tag/RabbitMQ

Jack works with me on the RabbitMQ core engineering team. We've been hard at work to address a lot of the issues brought up in comments here. It's worth it to try out our latest releases. The engineering team is very active with the community and takes all constructive, helpful (i.e. reproducible) feedback seriously. Feedback is encouraged via the rabbitmq-users mailing list. Thanks.

Post reply on HN