Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

Post reply on HN
Print view View original

71–80 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#71

Post by FridgeSeal »

raducu wrote: post #34
whalesalad wrote: post #30

Earlier quoted context omitted.

> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…

> afka is a code smell in a lot of organizations from my experience - most do not need it. Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.

These sound like consumer issues to me.

Kafka had been extremely reliable with latency, even under load in my experience.

If you’ve got badly lagging consumers that are trying to read from very old points in the topic while everyone else is at the head, you’ll definitely see some increased resource usage, but again, that’s mostly a consumer issue, and I’ve need seen performance degrades that much.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#72

Post by dalyons »

rafaelturk wrote: post #48

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

Rabbitmq sucks to scale. clustering and partitioning were terrible for a long time, maybe still is. Clusters dying in split brained ways, nodes crashing terribly / unrecoverable if they exceed iops or storage limits. You couldn’t pay me enough to run a high volume rMQ cluster again.

Never mind that the persistent durable log pattern of kakfa enables a lot replay type use cases that are very beneficial in financial systems specifically.

It’s not solely because of hype at all, it’s objectively better for many use cases.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#73

Post by leetbulb »

ceencee wrote: post #67
rafaelturk wrote: post #48

Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.

Pretty much every bank uses kafka ad the central messaging layer. What people are missing in almost every post here is the write once read many without data duplication and with different offsets is the killer app for Kafka beyond just the near infinite scale which is also super appealing. The failure modes are way way better than Rabbit as well. Note: I owned the streaming platform for a top 5 bank in the us.

Yeah, I'm sorry to others, but if you require the guarantees and compliance that Kafka provides, Kafka wins, especially at this kind of scale. I'd love to see RabbitMQ scaled out to handle hundreds of trillions of events per day and able to retain years worth of highly durable, immutable, and replayable event storage.

Ultimately, this comparison is apples vs oranges...

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#74

Post by zaphirplane »

supermatt wrote: post #18
eddythompson80 wrote: post #13

Earlier quoted context omitted.

I can't help but think that this just gives you the worst of both worlds. You are now on the hook managing that non-standard "smart" wrapper which will quickly just become the status quo for the project. Anyone wanting to change how it works needs to understand exactly how "smart" you made it and all the side effects that will come with making a change there. I pushed against knative in our company particularly for t…

Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.

mediocre Libraries sink to the bottom over time. Home grown libraries have a different quality and don’t sink easily. This is definitely YMMV for different teams

I can imagine home grown libraries having inconsistent api with wild and wonderful assumptions and beware the edge cases

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#75

Post by cbsmith »

robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

You're getting to the key thing:

You don't want to classify them by what they do. You want to classify them by what the clients must do/experience.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#76

Post by doctor_eval »

pnpnp wrote: post #69
robertlagrant wrote: post #4

> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…

Just my 2c but for anyone unaware, you should check out NATS. It combines the best of both Kafka and RabbitMQ IMO.

I thought NATS didn't actually store messages, am I mistaken?

Looking at Wikipedia (https://en.wikipedia.org/wiki/NATS_Messaging) I see that I'm technically right, it's JetStream that does the storage layer - but it's part of the NATS server.

From memory, I really liked the philosophy of NATS but found the nomenclature confusing.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#77

Post by dventimi »

> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also...

This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned transaction log" the author strands the reader without any help in absorbing the distinction.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#79

Post by bob1029 »

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network.

If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option.

If you are orchestrating the concerns of 5-10 services, I think you are making your life harder than it needs to be with ESB-style abstractions. Direct method invocation is much more reliable than whatever any one of these vendors could ever sell you. Put all your services into one exe. If you can't be bothered to use one language/repo, there are still ways to achieve this.

The real architect's dilemma is shoving one's edifice-constructing ego into a box long enough to produce a useful shack for the business.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#80

Post by cma256 »

The great thing about Kafka is the ability to batch operations. Collect a set of messages in memory and, when its time to commit, submit a bulk operation. If something fails you just rebuild the buffer from the last committed offset. Pretty neat piece of technology.
Post reply on HN