RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
eranstiller.com
RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
1–10 of 173 posts
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#2[deleted]
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#3The same MQ patterns as mentioned in the article (exactly once, consumer groups) can also be done in kafka, contrary to what the article suggests.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#4> one is a message broker, and the other is a distributed streaming platform
I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first 3 failures, as they could with RabbitMQ.
Having said that, Kafka lets you keep all your data, so you don't have to worry about losing messages to unexpected interactions between RabbitMQ rules. But having said that, now you have to store all your data.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#5If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#6My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind of an academic point no?
Unrelatedly, I've been looking at Pulsar lately. Anyone have experience with Pulsar and either RMQ/Kafka want to throw out some opinions from having tried both?
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#7NATS (https://nats.io/) is another option, though I'm not sure if it's still considered a viable Kafka replacement.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#8> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…
> One is smart messaging; dumb clients. The other is dumb messaging; smart clients.
All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients.
The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#9Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#10I think this is an excellent article. The only thing I'd add is that RabbitMQ is an implementation of AQMP (optionally v1.0) as a standardized broker service protocol so is designed to be interchangeable with other extant implementations such as Apache Active MQ and Cupid whereas Kafka is one-of-a-kind software. Beyond that RabbitMQ has standardized client libs and frameworks in Java land if that matters to you - it did matter in the original context of message queue middlewares and SOA from where AMQP originated and where enterprise messaging sees major use. OTOH Kafka, with caveats, is in principle more "web scale" - though that is far from a free ride.