Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

41–50 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#41

Post by KaiserPro »

speedgoose wrote: post #35
fuzzy2 wrote: post #19

Earlier quoted context omitted.

NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources. I'd recommend considering it, especially as an alternative to RabbitMQ.

I only tested NATS using JetStream and I struggled with the throughput in Python. I probably used it wrong. But your comment may imply that jetstream is slow.

we used it for some low latency stuff in python. it was about 10ms to enqueue at worst. However we were using raw NATS, and had a clear SLA that meant that the queue was allowed to be offline or messages lost, so long as we notified the right services.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#42

Post by math »

raducu wrote: post #26
robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…

Worth noting that Kafka is getting queues: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#43

Post by FINDarkside »

JohnMakin wrote: post #38
whalesalad wrote: post #33

Earlier quoted context omitted.

It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)

No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.

So let me get this straight. You've used Kafka once, RabbitMQ never. You don't really know what you did with Kafka. But you somehow know that RabbitMQ cannot do the thing which you don't really remember anymore. Doesn't make much sense to be honest.

Nobody can really have any sources for RabbitMQ being able to do it if you don't know what it supposedly cannot do. The way you descibed it, is that you simply read data and then did something with the data and passed it to somewhere else. RabbitMQ obviously can do it.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#45

Post by KaiserPro »

FridgeSeal wrote: post #22

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…

> now you can also replay queues

yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong)

its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for.

Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it can scale, but also it can just sulk.

rabbit has it's own set of problems, and frankly it's probably not choose either anymore.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#46

Post by nhumrich »

klabb3 wrote: post #27
dividedbyzero wrote: post #7

NATS ( https://nats.io/ ) is another option, though I'm not sure if it's still considered a viable Kafka replacement.

It’s true FOSS, and the server is standalone Go binary that’s so small it can even be embedded. Lots of language bindings for clients. Has persistence, durability, and nicely aligns into a raft-like cluster in a DC without a separate orchestrator. I’m a big fan – never understood why it’s not at the top of the list in these tech reviews.

Rabbitmq is FOSS, has lots of language bindings. It has persistence, durability, and doesn't require a separate orchestrator.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#47

Post by supermatt »

JohnMakin wrote: post #38
whalesalad wrote: post #33

Earlier quoted context omitted.

It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)

No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.

> Can’t find anything that suggests rabbitmq natively supports anything like sink connectors

Kafka doesnt natively support them either. That would be Kafka Connect. I guess you could use it as an MQ, but it wouldnt be a very good one. Its more used as a data integration platform. If you want more MQ-like functionality OOTB on top of Kafka you would want to use something like Kafka Streams instead.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#48

Post by rafaelturk »

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#49

Post by nhumrich »

KaiserPro wrote: post #37
whalesalad wrote: post #16

One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…

Rabbit is an arse to scale past one broker. It was possible, but a pain, that might have changed. Kafka is just a pain full stop.

The official rabbitmq controller for kubernetes is a breeze. Scales wonderfully without almost any effort.
Post reply on HN