Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

61–70 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#61

Post by officialchicken »

KaiserPro wrote: post #45
FridgeSeal wrote: post #22

I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…

> now you can also replay queues yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong) its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for. Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it ca…

> (I'm looking at you new york times, you know what you did wrong)

You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#62

Post by lmm »

ryanjshaw wrote: post #55
raducu wrote: post #26

Earlier quoted context omitted.

> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…

> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consu…

> You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.

And there are libraries that will manage all this for you e.g. https://github.com/line/decaton

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#63

Post by esafak »

bhouston wrote: post #5

If you use Confluent Kafka, the billing is pretty high. About 4 years ago it was much cheaper, but then they completely revamped the pricing to something ridiculous. I found that switching to Google Pub/Sub, at least if it meets your needs, is cheaper.

I see they offer Kafka's exactly-once delivery: https://cloud.google.com/blog/products/data-analytics/cloud-...

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#64

Post by facorreia »

math wrote: post #42
raducu wrote: post #26

Earlier quoted context omitted.

> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…

Worth noting that Kafka is getting queues: https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

And also Rabbit has streams[1]. There's a lot of overlap.

[1] https://www.rabbitmq.com/streams.html

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#65

Post by hamandcheese »

supermatt wrote: post #18
eddythompson80 wrote: post #13

Earlier quoted context omitted.

I can't help but think that this just gives you the worst of both worlds. You are now on the hook managing that non-standard "smart" wrapper which will quickly just become the status quo for the project. Anyone wanting to change how it works needs to understand exactly how "smart" you made it and all the side effects that will come with making a change there. I pushed against knative in our company particularly for t…

Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.

> Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that?

This is decent advice, IMO. The cost of dependency management is often vastly understated.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#66

Post by EdwardDiego »

BWStearns wrote: post #6

My comment is mostly about part 2 of this post, but wrt message ordering being a kafka "win" I'd raise the point that in the actual use case of "a consumer fails in some way to process the message" you can still end up with out of order processing of the consumer's input since you might want to dump them into a DLQ or something. The fact that the message isn't reappended to the topic by default for processing is kind…

Pulsar can have both MQ semantics and pub/sub semantics. In pub/sub it's sorta like "Kafka with all bits people found it necessary to build later already built in", e.g., a proxy, schema registry, connectors, replication, tiered storage, all out of the box.

It also has lightweight streaming functions built-in, but they operate per record, so good for lightweight transforms/routing, not for stream aggregations etc.

It has more moving parts also, brokers are decoupled from storage, which is handled by BookKeeper, and replication between two clusters requires a ZooKeeper that both clusters can access, in addition to the ZK used by the brokers and bookies.

And it's a reasonably new project, so last time I looked into it, some of the documentation was incorrect, especially around managing bookies.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#67

Post by ceencee »

rafaelturk wrote: post #48

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

Pretty much every bank uses kafka ad the central messaging layer. What people are missing in almost every post here is the write once read many without data duplication and with different offsets is the killer app for Kafka beyond just the near infinite scale which is also super appealing. The failure modes are way way better than Rabbit as well. Note: I owned the streaming platform for a top 5 bank in the us.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#68

Post by ceencee »

rafaelturk wrote: post #53
rafaelturk wrote: post #48

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.

This is a ridiculous statement if you really build an EDA. Kafka is what enables the decoupling.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#69

Post by pnpnp »

robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

Just my 2c but for anyone unaware, you should check out NATS.

It combines the best of both Kafka and RabbitMQ IMO.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#70

Post by ceencee »

raducu wrote: post #34
whalesalad wrote: post #30

Earlier quoted context omitted.

> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…

> afka is a code smell in a lot of organizations from my experience - most do not need it. Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.

Kafka can sustain sub 20ms at millions or even billions per second scale. Processing time delays is bad consumer code and partition design smell. Aka , your consumer shouldnt depend on a slower resource within an ordering domain. This can also be mitigated with an async consumer
Post reply on HN