MQTT vs. Kafka: An IoT Advocate's Perspective

You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress.

Example - you have 1 topic with 1 producer and 20M consumers. Each consumer is a tcp connection. Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers, 100M connected clients and your connections are randomly distributed over your brokers. You don't want to route every message to every broker because. So you need a fat pipe and some middle man that knows which brokers a message must be routed to because there are currently consumers connected to those brokers waiting subscribed to topics and waiting for messages. As someone who works on MQTT, you for sure understand the problem.

I have never heard of Eclipse Amlen. However, I am working with MQTT and Kafka since 2012 and have seen a nation-wide successful MQTT rollouts where MQTT and Kafka worked in tandem to solve exactly the problem you are talking about - millions of little concurrent connections distributed over a large fleet of devices for sub-100ms round trip.

It's not a competition, it's a coopoeration.

The vast majority of "Kafka problems" (or rather "Kafka Streams problems") we have with a managed solution at work are due to not fully understanding how it works and how to set it up. There's so much stuff you can configure and so much potential to misuse it. Typical problems are wrong configuration for acks, not understanding durability guarantees, not understanding exactly-once semantics, not naming hidden topics in Kafka Streams, not using schema registry serializers for hidden topics, choosing the wrong partitioning scheme (and wanting to change it later), using Kafka clients with different hash functions, using wrong schema compatibility mode, etc. etc.

> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed?

When the use case warrants it and when downstream consumers need to be notified of a change in the data or its state. That is, in reactive architectures.

A data streaming platform (of which Kafka is one example) will push the data, whereas a database (relational or not) requires to be explicitly polled (queried)[1]. A poll for new data is inefficient, is computationally expensive (a waste of CPU cycles when the data has not changed) and is prone to create creating delays in the data processing (when the polling interval is too long). Data streaming platforms will deliver the changes in near realtime.

Also, Kafka, being a distributed append only log, keeps an entire temporal history of all data changes. That is, if an event or an object has undergone a series of state transitions, such as create -> update 1 -> update 2 -> update 3 -> delete, it will be fully available in a topic as five separate events. If a consumer is only interested in the latest state change, a series of events can be compacted into an equivalent of a database table (naturally, it is called a K-table in Kafka) and the consumer can use the K-table instead of having to process the entire event stream[2]. K-tables and event streams can be flipped over with no extra effort.

Kafka is, effectively, your databases's own transaction log sans the query engine.

> Why do you need a sharded Kafka cluster?

Kafka cluster are not sharded. At least not in the database sense, as it is a cluster. A consumer sees one living thing[3] – the cluster, and the cluster can be dynamically resized without the consumer(s) knowing or noticing it. The cluster will then rebalance itself.

> Where/why add Kafka to that stack?

Kafka is typically used to get the data out of data sources and deliver a temporal series of changes to interested consumers. If the data needs to queried ad-hoc or explicitly, Kafka is then fronted by one or many (usually, domain specific) event stores. An event store can be a database (a RDBMS or a document one), a document search engine (e.g. Elastic Search or a vector database) or anything else that has a query engine in it. It can be a cache as well when either the speed of retrieval or affinity to the data are important.

[1] It is possible to mimic the Kafka temporal history via the use of triggers that would inject a copy of the row being changed into an «audit» table before allowing a update to occur, but, with time and in high data volume environments, the table size will balloon and will drag the database down.

[2] Most databases now have change event streams that abstract the database transaction log away and deliver raw row changes as a temporal history of events, which is a functional equivalent of data/event streaming (using Kafka or similar) sans the scalability and the distributed processing – the onus to implement either or both is entirely on the consumer and that part is hard. Therefore, a database change event stream is oftentimes hooked directly into a Kafka cluster.

[3] Not entire true in most implementations, but it is rabbit hole for another time to go down into.

Architecturally AMQP is just bigger brother of MQTT, to the point some queue brokers (like RabbitMQ) can accept both.

MQTT 5 made difference smaller too.