many people seem to not have clarity on what a distributed log is and in which architecture its useful and in which not.
if you are abusing a distributed log as a message queue, you are most of the time creating a mess.
MQTT vs. Kafka: An IoT Advocate's Perspective
31–40 of 122 posts
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#32The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us…
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#33Earlier quoted context omitted.
> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us…
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
Maybe something AMQP related instead?
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#34many people seem to not have clarity on what a distributed log is and in which architecture its useful and in which not. if you are abusing a distributed log as a message queue, you are most of the time creating a mess.
The "abuse" you describe is pretty much how Kafka is used everywhere I've worked.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#35An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
What issues did you run into? From a technology perspective it's been rock solid for years in my experience. Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the…
Do you know of any places to learn those things? Kafka seems pretty interesting to me
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#36The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
> implying the operational costs of a server are captured in its per hour sticker price
managed kafka has been around a while
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#37Earlier quoted context omitted.
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
Maybe something AMQP related instead?
Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two.
Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message.
Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine.
Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator.
If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#38The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
Forget MQTT, Redis as a PUB/SUB will do 99% of the job most of the time.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#39I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.
With mqtt it depends on the broker, eg. Emitter.io can save them for a week etc.
Offset for a client is usually stored on the broker, so if a client reconnects, all of the messages it has missed are forwarded to it.
As mentioned in other answers, service levels have a defined meaning, which is different to the absolute theoretical meaning, and really is to do with the message aks.
Really, as the article mentions, kafka and mqtt are for different purposes, with some overlap. Kafka is all about the log, whereas mqtt is about uncertain connections. A better comparison which I've yet to see, is comparing mqtt to nats.
Lastly, kafka is much easier to administer using redpanda, which doesn't have zookeeper, combines the registry and kafka connect (see WASM runners) with the runtime, and has a very nice console for debugging.
Similarly, Emitter.io does a great job with clustering for mqtt.
I'd like to see an open source kafka-mqtt bridge that worked in both directions as they all seem to go mqtt->kafka only.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#40I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.
MQTT isn’t designed as a persistent log, but can fulfil some of what you might want to use one for.
Each message has a couple of flags, the first being Quality of Service, which as you quoted above determines deliver guarantees. 0 is fire and forget, with potential loss of messages. 1 will queue messages for delivery to offline clients that are subscribed to a topic (within reason, all brokers set limits on that), and 2 is often described as “exactly once”, but is in fact just a more involved dance to acknowledge messages.
The other flag is a Retain flag, which instructs the broker to associate that message with the topic it was sent to, and send it on to any newly subscribing clients when they subscribe. This is good for use cases like remote device configuration - you can send it to a topic, setting the retain flag, and then when a device comes online it’ll immediately receive new configuration.
MQTT is great as a message queue for remote devices, mostly because it’s so lightweight anything with an IP stack can integrate with it, but I’m not sure why anyone would attempt to make it a piece of core infrastructure.