An introduction to RabbitMQ • Hacker News Board

Re: An introduction to RabbitMQ

#21

Post by dgritsko » Thu, May 21, 2020, 3:19 PM UTC

FlorianRappl wrote: post #6

Using it in multiple projects: The software itself is great and provides great value. Only pitfall are the available libs. Especially with the .NET implementation we had quite a lot of trouble. Its not following current .NET patterns and has strange quirks. Does anyone know a good alternative to the "official" one?

It's been a few years since I used it, but I used EasyNetQ for a while when I was working with RabbitMQ and it was great. A quick peek at GitHub shows that it still seems to be actively maintained. Maybe it's what you're looking for: https://github.com/EasyNetQ/EasyNetQ

Re: An introduction to RabbitMQ

#22

Post by csharptwdec19 » Thu, May 21, 2020, 3:19 PM UTC

FlorianRappl wrote: post #6

Using it in multiple projects: The software itself is great and provides great value. Only pitfall are the available libs. Especially with the .NET implementation we had quite a lot of trouble. Its not following current .NET patterns and has strange quirks. Does anyone know a good alternative to the "official" one?

You might want to look into EasyNetQ[0]. I've not played with it much but it appears to be a cleaner, more modern abstraction over the existing .NET Client. I'm not sure whether it fixes all the 'quirks' in the client however (I've run into them too :))

[0] - https://github.com/EasyNetQ/EasyNetQ

Re: An introduction to RabbitMQ

#23

Post by evdev » Thu, May 21, 2020, 3:20 PM UTC

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

Rabbit MQ is a traditional message broker; you use it when you have lots of messages you don't particularly want/need to be stored persistently, and where you want/need to take advantage of the routing feature--that you put keyed messages into some topic/exchange and then subscribe to only part of the messages any given application is interested in.

Kafka creates the abstraction of a persistently stored, offset-indexed log of events. You read all events in a topic. Kafka can be used to distribute messages in the way AMQP is used, but is more likely to be the centerpiece of an architecture for your entire system where system state is pushed forward/transformed by deterministically processing the event logs.

Re: An introduction to RabbitMQ

#24

Post by spamizbad » Thu, May 21, 2020, 3:21 PM UTC

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

If your main concern is scalability: Each queue in rabbit gets its own thread. So if you can spread your workload across multiple different queues you can scale without too many problems.

Re: An introduction to RabbitMQ

#25

Post by pc86 » Thu, May 21, 2020, 3:22 PM UTC

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

Are you replacing an existing system that's already at scale?

Re: An introduction to RabbitMQ

#26

Post by dirtydroog » Thu, May 21, 2020, 3:22 PM UTC

We use ZeroMQ a bit. It's been pretty much flawless as far as I can see but I get the impression that it's becoming obsolete. Is RabbitMQ a viable replacement?

Re: An introduction to RabbitMQ

#27

Post by pc86 » Thu, May 21, 2020, 3:24 PM UTC

bvm wrote: post #15

Using this opportunity to shout out to Rascal ( https://github.com/guidesmiths/rascal ) which makes using RabbitMQ on Node an absolute joy.

Same with MassTransit[0] and .NET. We have several distributed .NET Core services running in our data center, services running on employee PCs, etc all communicating via RMQ with MassTransit and it's great. The primary maintainer is very active (streams every Thursday evening) and the documentation has gone from "pretty bad" to really good in the last few months.

[0] https://masstransit-project.com/

Re: An introduction to RabbitMQ

#28

Post by halfmatthalfcat » Thu, May 21, 2020, 3:24 PM UTC

pc86 wrote: post #25

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

Are you replacing an existing system that's already at scale?

No, greenfield

Re: An introduction to RabbitMQ

#29

Post by vel0city » Thu, May 21, 2020, 3:24 PM UTC

halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

I've been running RabbitMQ on pretty small VMs for a long time. RabbitMQ doesn't need a lot of resources per message, even with very small VMs (512MB RAM, single CPU) I've seen it handle peaks of many thousands of messages a second without running into problems. Give it a bit of beefy hardware and it'll probably handle whatever load you were thinking, unless you're saturating 10gig links with messages or something.

RabbitMQ and Kafka are very different struggles when thinking of scaling and performance. Kafka is almost a database itself of messages which have routed through the system. In many configurations clients can come back and demand to replay the message stream from almost any point in time. This means you need to handle _a lot_ of disk and memory access. With RabbitMQ, messages are traditionally very ephemeral. Once a message has been ack'd, its gone. Poof. Not in memory. Not on disk. Nobody is going to come back asking for that message. This leads to a lot more efficiency in handling things per message, but at the cost of not being able to remember the messages that went through the system a few milliseconds ago.

Re: An introduction to RabbitMQ

#30

Post by eternalban » Thu, May 21, 2020, 3:25 PM UTC

posharma wrote: post #10

How does RabbitMQ compare with Kafka?

RabbitMQ is not suitable for event sourcing. Kafka is. In general, RabbitMQ is a “river” and Kafka is a “lake”.

RabbitMQ has excellent support for complex message flow topologies. Kafka out of the box does not provide these features.