Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

251–260 of 263 posts

Re: An introduction to RabbitMQ

#251

Post by lukebakken »

adamkf wrote: post #191

I've had truly terrible experiences with RabbitMQ. I believe that it should not be used in any application where message loss is not acceptable. Its two big problems are that it cannot tolerate network partitions (reason enough to never use it in production systems, see https://twitter.com/antifuchs/status/735628465924243456 ), and it provides no backpressure to producers when it starts running out of memory. In my l…

Kyle's analysis of RabbitMQ is almost 6 years old. Rest assured, things have changed since then.

Re: An introduction to RabbitMQ

#252

Post by shaklee3 »

jonathanoliver wrote: post #249
shaklee3 wrote: post #247

Earlier quoted context omitted.

This isn't true anymore. Nats streaming has persistence, so the OP's question still remains

My understanding is that NATS (a protocol) and NATS streaming were related but separate: https://github.com/nats-io/nats-site/issues/217 (The issue is from 2017 but illustrates a distinction)

That's right, but I think at least since both are listed on their website as different ways to run it that it should at least be considered a native feature at this point.

Re: An introduction to RabbitMQ

#253

Post by carterklein13 »

shaklee3 wrote: post #252
jonathanoliver wrote: post #249

Earlier quoted context omitted.

My understanding is that NATS (a protocol) and NATS streaming were related but separate: https://github.com/nats-io/nats-site/issues/217 (The issue is from 2017 but illustrates a distinction)

That's right, but I think at least since both are listed on their website as different ways to run it that it should at least be considered a native feature at this point.

All very interesting - this is great!

Re: An introduction to RabbitMQ

#254

Post by MR4D »

emmelaich wrote: post #132
MR4D wrote: post #73

Earlier quoted context omitted.

> The only downside is once you get message-queue-pilled, ... I think this is why email will never die. It's basically turned into a huge message queue. Even voice mails come into my inbox. ====== EDIT - I meant to say "huge universal message queue" and left out the word "universal" accidentally

And you can literally use the maildir format for a queue! https://pypi.org/project/dirq/ Perl had the original implementation, and there are implementations in other languages.

I forgot about that - used to be a cool hack!

Re: An introduction to RabbitMQ

#255

Post by theptip »

hrpnk wrote: post #216
theptip wrote: post #139

Earlier quoted context omitted.

In my experience RMQ is solid enough that for many use-cases it's reasonable to run it without a standby (especially if you're on Kubernetes where you'll get a replacement instance created automatically if your active instance fails). A common use-case is for async tasks (Celery) that can tolerate a few minutes of downtime. If you're running a fully evented architecture then this might not apply - though if you're no…

Which Kubernetes operator for RabbitMQ are you using?

I’m just using the Helm manifests, but when I set this up operators were not a thing. I’d probably look into the operator approach if I was starting from scratch now.

Re: An introduction to RabbitMQ

#256

Post by kchr »

jv22222 wrote: post #164
webscalist wrote: post #117

RabbitMQ has huge learning curve if you're trying to build a worker queue. First, you'll learn about ack/noack and get the worker ack on success. Then, you'll learn about dead letter queue ... etc for delayed retries. Now, you'll have a topic exchange and a bit hairy routing in place using wildcards. And you mistakenly set dead letter routing key so that expired messages end up in multiple queues (retry queues and ac…

The biggest annoyance I found with RabbitMQ was that it could take up to 10-15 mins to restart if it had a lot of jobs. This was back in 2015 - might be better now.

Sounds like resource (design) issues to me.

Re: An introduction to RabbitMQ

#257

Post by mcsoft »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

We used both zeromq and its sort-of-descendant nanomsg but eventually moved to NATS because of it's approach to high availability (every node knows the topology of the whole cluster) and zero management overhead.

Some criticize NATS for the absence of message durability in the core technology, but we figured out we can drop this requirement in 95% of cases. Your microservices should be highly available anyway, so there's always a live consumer - and it's better to handle the message to it directly rather than introducing the overhead of storing a message somewhere and dealing with more-than-once delivery.

There's a bit more to that like you should let your microservices exit gracefully and finish processing consumed messages. And of course in 5% of cases, where you can't allow to lose a single message, you have to use NATS Streaming or the likes, but so far we've been greatly impressed by NATS for high load.

Re: An introduction to RabbitMQ

#258

Post by mcsoft »

ies7 wrote: post #171
mcsoft wrote: post #145

Earlier quoted context omitted.

http://nats.io

I heard a lot of praise about Nats, but isn't it more like a kafka alternative? Someone new need to spend sometime grasping the stream concept.

One practical reason we chose Nats over Kafka was that Nats doesn't need zookeeper for HA.

Nats doesn't provide message durability too, luckily it's not required for 95% of our use cases. Also, having NATS already implemented it's a natural move to use NATS Streaming for durability rather than introducing a completely new technology to your stack.

Less pieces - fewer chances something breaks.

Re: An introduction to RabbitMQ

#259

Post by AmericanChopper »

tracker1 wrote: post #239
AmericanChopper wrote: post #152

Earlier quoted context omitted.

I understand where you're coming from, but what you're describing is learning how to use a queue to maintain consistency guarantees across a distributed system. You can get something simple like AWS SQS working with a few clicks, but then you don't have any of those consistency guarantees.

If you don't need crazy throughput, I find that Azure Storage Queues are crazy easy, built in retry and just simple as can be. Though when I've used it in the past, I've created a slightly simpler to use abstraction. https://docs.microsoft.com/en-us/azure/storage/queues/storag... Thinking of doing something that works like an async generator so I can just use it like... const work = queue.subscribe('somequeue'); for…

Azure Storage Queues are about on par with SQS. That is, easy to use, but lacking strict concurrency control. If you need that level of concurrency control (and stricter serialization), then you’d be better off their (more complicated) service bus product [0].

RabbitMQ isn’t more complicated because it’s been improperly designed. It’s more complicated because it’s doing a much more complicated task.

A fair amount of that complexity lies in the hosting, so a managed service can take some of that off your hands (for an increased price obviously), but part of it is necessarily going to lie with the message consumer (your application logic). If your use case doesn’t need that level of control, then it doesn’t need that level of complexity either, so something like Rabbit would just be the wrong choice.

[0]: https://docs.microsoft.com/en-us/azure/service-bus-messaging...

Re: An introduction to RabbitMQ

#260

Post by nathan-io »

kabacha wrote: post #177

I'm really surprised to see so much positivity about rabbitmq here when it's probably the most sweared-at software in the space. Let me share my anecdote. In my last work place I got onboarded on rabbitmq and it was such a painful software to work with and almost impossible to set up locally that I silently sneaked in simple redis list as queue alternative for my dev environment. The whole rabbitmq and it's pika libr…

I'm equally surprised by your anecdote.

I found it quite simple to install RabbitMQ server and its admin panel in my WSL local dev environment.

And the cloud/prod instance took a few clicks (just spun up a DO Marketplace server image) followed by It was also dead simple to start using RabbitMQ within my application. I found a well maintained package, installed it, edited a couple lines of my application's config, and everything just worked.

I specifically avoided Redis based on my understanding that it can't guarantee message persistence, so if it crashes, your unprocessed messages are lost.

Post reply on HN