> Apache Kafka is not an implementation of a message broker. Instead, it is a distributed streaming platform. Unlike RabbitMQ, which is based on queues and exchanges, Kafka’s storage layer is implemented using a partitioned transaction log. Kafka also... This seems like an important passage, drawing the crucial and long-awaited distinction between RabbitMQ and Kafka, and yet without having defined a "partitioned tran…
RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
121–130 of 173 posts
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#122Thank you for all the feedback! Given the advances in the past few years in this area, this article could use a serious update. A comparison to NATS and Pulsar is warranted, along with some extra explanations for some of the technical terms in the article.
I'll carefully review the feedback here and try to make an update at some point soon.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#123Earlier quoted context omitted.
Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? Its just an abstraction layer to make things less effort.
> Thats kind of like saying dont use any software libraries because they all use the standard lib indirectly so you may as well just use that? This is decent advice, IMO. The cost of dependency management is often vastly understated.
The cost of dependency management may be understated but it’s always less than the cost of reimplementing everything found in established libraries.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#124Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#125I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…
This is a SOLVED usecase.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#126Wow, this is the type of trash one can expect from someone calling themselves "Architect"
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#127Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#128RabbitMQ is one the most impressive technology I have ever used I think. It's simple, efficient and reliable. In 10 years, I have never seen a crash or a bug, even with heavy load.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#129The simplest thing you can do is to realise WHY you are using messages. Messages are there to trigger a process. Usually, you don't need a lot of data to trigger a process, the message just needs to let the system know enough to locate all necessary information.
Also, when you are sending information at an extremely high rate, there usually is no difference if each message is processed separately or in batches.
So what you can do in practice?
1) Get the producer to batch the messages. For example set rules like "batch up to 10.000 messages, up to 100ms, up to 100MB of data, whichever comes first". 2) Serialise the batch (for example, if it makes sense, create a compressed JSON file) 3) Upload the file to some high throughput, scalable, cheap storage (for example S3) 4) Send a message to the queue / topic / whatever else you are using with just enough to locate and process the message -- usually just the link to the S3 object.
This usually can be modified depending on specific project needs.
Now your messaging only ever sees a small number of very small messages and you will never have any scaling problems, at least not on messaging side.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#130Earlier quoted context omitted.
Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.
My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.