Live data from Hacker News

RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

eranstiller.com

121–130 of 173 posts

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#121

> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also... This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned tran…

Hi, this is the article author here. Thanks for the feedback! I've written this article 3.5 years ago, and it could definitely deserve a shake-up. I agree this should be cleared up a bit.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#122
Hi everyone, this is the article author here. I was genuinely surprised to see the article pop up here 3.5 years after I wrote it.

Thank you for all the feedback! Given the advances in the past few years in this area, this article could use a serious update. A comparison to NATS and Pulsar is warranted, along with some extra explanations for some of the technical terms in the article.

I'll carefully review the feedback here and try to make an update at some point soon.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#123

Earlier quoted context omitted.

Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.

> Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? This is decent advice, IMO. The cost of dependency management is often vastly understated.

That’s Not Invented Here syndrome, and it’s decidedly bad advice.

The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#125
post #79

I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…

Your usecase is EASY - its EASY!

This is a SOLVED usecase.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#128
post #127

RabbitMQ is one the most impressive technology I have ever used I think. It's simple, efficient and reliable. In 10 years, I have never seen a crash or a bug, even with heavy load.

where i work we have some queues with millions of items in and the thing is perfectly stable.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#129
If you have a throughput problem you are most likely doing it wrong. If your knee jerk reaction is to scale your messaging up you may want to reconsider. Messaging systems are usually hard to scale up and always very costly to do so compared to the amount of data they are transferring.

The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information.

Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches.

So what you can do in practice?

1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first". 2) Serialise the batch (for example, if it makes sense, create a compressed JSON file) 3) Upload the file to some high throughput, scalable, cheap storage (for example S3) 4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object.

This usually can be modified depending on specific project needs.

Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.

Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

#130

Earlier quoted context omitted.

Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.

My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.

I have to ask, what value is this adding business-wise to store so much?
Post reply on HN