Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

81–90 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#81
If you want features of RabbitMQ (specifically queue like behavior) but the scalability of Kafka then you probably want Apache Pulsar.

To elaborate on that a bit the main things Pulsar gives you are:

1. Still underlying distributed stream based architecture, this is what makes it able to do Kafka like things.

2. Broker side management of subscription state which allow out of order acknowledgement, this means you can use it like a queue. (Subscriptions sort of act like AMQP mailboxes but without the exchange routing semantics). Vs Kafka which can only do cumulative acknowledgement, i.e head of line blocking.

3. Separated "compute" and storage. By storing data in Bookkeeper you can scale your needs to support a lot of consumers separately from how you stash the data those consumers need to read vs Kafka where these 2 are coupled and an imbalance between the two becomes awkward.

4. In built offload with transparent pass-through read. When your data falls off the retention cliff for your standard broker cluster the data can be archived to object storage. The broker can transparently handle read request for these earlier messages though, just with higher startup latency to pull the archived ledgers.

5. Way more plugability than Kafka, in-fact similar plugability as RabbitMQ. You can implement your own authz/authn, a different listener to support a different protocol (there is a Kafka one, MQTT, AMQP etc).

6. Much greater metadata scalability. Before the new KRaft implementation the layout of metadata in ZK meant that you couldn't feasibly have more than about 10k topics. Especially because of how long the downtime would be on controller failover. Pulsar can easily support much larger numbers of topics which prevents needing to use a firehose design when you would prefer individual topics per tenant/customer/whatever.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#82
post #34

Earlier quoted context omitted.

> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…

> afka is a code smell in a lot of organizations from my experience - most do not need it. Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.

If you're concerned about latency you might want to consider zeromq. Stream processing doesn't really have a time expectation to it.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#84
post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

Or NATS

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#85

One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…

>> Rabbit can do everything Kafka does - and much more - in a more configurable manner. Sure, if you're doing like 10's of MB/s. RMQ is fast compared to AK if you're not adding durability, persistence, etc. Try to run gigabytes per second through it though, or stretch across regions, or meet RTO when the broker gets overloaded and crashes.. Get your shovel ready! ;)

Kafka itself is dumb but scalable and resilient, it's the client ecosystem that's massive compared to RabbitMQ. Count 10 stream processing, connectivity, ingestion or log harvesting platforms that use RMQ as it's backend, then name 10 languages that have supported libraries for RMQ.. then compare that to Kafka.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#86

My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind…

Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box. It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc. It…

Massive and complex platform.. at a certain point why not just run 2 different platforms that are best of breed for each.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#87

Redis isn't an option?

Redis is a cache, it's not a queue. Can you do something similar? I guess, just like you can use a highlighter to paint your house. But the semantics are not correct.

There is Redis Streams, but certainly not without it's problems. Super obscure, not a lot of client support.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#88
post #5

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

It's cheaper until you get to a sizeable workload, and the P90+ latency is ridiculous.. the kafka api is weak and when you're not using kafka api you're limited on integration tools unless you want to be super locked in to GCP.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#89
post #63
post #5

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

I see they offer Kafka's exactly-once delivery: https://cloud.google.com/blog/products/data-analytics/cloud-...

Kafka doesn't guarantee exactly once delivery at all, unless you're using Kafka Streams and even then your final output topic still won't get exactly once, the consumer group protocol doesn't allow for it.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#90
post #5

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

Yes, I can confirm that. Confluent is the most expensive part of our current infrastructure.

Sounds about right.
Post reply on HN