RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1) • Hacker News Board

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#21

Post by whalesalad » Tue, Sep 19, 2023, 9:00 PM UTC

JohnMakin wrote: post #17

whalesalad wrote: post #16

One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…

I am not an expert in either and have only worked with Kafka. At a past job I had to write a connector job to parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else. RabbitMQ supports this? What is the one use case of kafka? I think you have it backwards.

> parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else

can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#22

Post by FridgeSeal » Tue, Sep 19, 2023, 9:01 PM UTC

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design.

You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling properties-which doesn’t hurt at the small scale, and provides further scaling when you need it.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#23

Post by eddythompson80 » Tue, Sep 19, 2023, 9:03 PM UTC

supermatt wrote: post #18

eddythompson80 wrote: post #13

Earlier quoted context omitted.

I can't help but think that this just gives you the worst of both worlds. You are now on the hook managing that non-standard "smart" wrapper which will quickly just become the status quo for the project. Anyone wanting to change how it works needs to understand exactly how "smart" you made it and all the side effects that will come with making a change there. I pushed against knative in our company particularly for t…

Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.

yeah, don't wrap all calls to a standard lib in another homegrown or non-standard single-digit user lib that makes changes in all sort of subtle ways. There are plenty of C++ projects that make their own or wrap stdlib and they are always a big wtf.

It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client that reimplements something like rabbitmq, and much worse a smart service.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#24

Post by supermatt » Tue, Sep 19, 2023, 9:05 PM UTC

eddythompson80 wrote: post #23

supermatt wrote: post #18

Earlier quoted context omitted.

Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.

yeah, don't wrap all calls to a standard lib in another homegrown or non-standard single-digit user lib that makes changes in all sort of subtle ways. There are plenty of C++ projects that make their own or wrap stdlib and they are always a big wtf. It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client that reimplements something like rabbitmq, and much worse…

> don't wrap all calls to a standard lib

Im not saying to expose the same primitives - what would be the point in that? I am saying that EVERY lib you use will be using the standard lib or some abstraction of it to perform its own utility.

> It's one thing to have an abstraction for kafka in your code, it's another to wrap the client in a smart client, and much worse a smart service.

That abstraction is exactly what i am talking about. Why write 50 lines of boilerplate multiple times throughout your code when you can wrap that up in a single function call and expose THAT as your client. You know thats exactly what you will end up doing on any non-trivial project. Or you could use a lib that already does that - such as the "official" kafka streams lib.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#25

Post by JohnMakin » Tue, Sep 19, 2023, 9:05 PM UTC

whalesalad wrote: post #21

JohnMakin wrote: post #17

Earlier quoted context omitted.

I am not an expert in either and have only worked with Kafka. At a past job I had to write a connector job to parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else. RabbitMQ supports this? What is the one use case of kafka? I think you have it backwards.

> parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.

It was a Sink Connector. I don’t know what it was or wasn’t supposed to do but I was asked to do it, as is often the case in tech. I could have done any number of transformations in that process though, which I’m not sure rabbitmq supports

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#26

Post by raducu » Tue, Sep 19, 2023, 9:07 PM UTC

robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

> who can't just now pluck messages off a queue to process

The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.

If a certain message processing takes very long, all other messages in that partition will have to wait.

Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is much higher; but you do get nice message ordering for any given partition in kafka which you do not get in RabbitMq (afik); you are also get some really nice data locality with kafka because unless the consumers get the partitions re-assigned, all messages for the same key are served on the same physical consumer.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#27

Post by klabb3 » Tue, Sep 19, 2023, 9:07 PM UTC

dividedbyzero wrote: post #7

NATS ( https://nats.io/ ) is another option, though I'm not sure if it's still considered a viable Kafka replacement.

It’s true FOSS, and the server is standalone Go binary that’s so small it can even be embedded. Lots of language bindings for clients. Has persistence, durability, and nicely aligns into a raft-like cluster in a DC without a separate orchestrator.

I’m a big fan – never understood why it’s not at the top of the list in these tech reviews.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#28

Post by [deleted] » Tue, Sep 19, 2023, 9:10 PM UTC

waynesonfire wrote: post #12

supermatt wrote: post #8

Earlier quoted context omitted.

> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.

And reimplement rabbitmq? Great idea. Let's do it in rust too.

[deleted]

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#29

Post by klabb3 » Tue, Sep 19, 2023, 9:10 PM UTC

bicijay wrote: post #9

Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.

There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#30

Post by whalesalad » Tue, Sep 19, 2023, 9:11 PM UTC

FridgeSeal wrote: post #22

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…

> You can get all the same advantages of message acknowledgments, but now you can also replay queues

with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do.

> let different applications use the messages (handy for cross cutting event/notification systems)

rmq also does a publish once and fanout to multiple queues to support this. data is replicated so that could be a deal breaker, but it is possible.

how often have you had to diagnose a stuck consumer or some other kind of offset glitch where a consumer is unable to resume where it left off?

not knocking kafka here but I do think it is a tool you should reach for when you need to solve a very hyper focused problem, while rabbit is a tool more suited to most cases where queuing is required. kafka is a code smell in a lot of organizations from my experience - most do not need it.