Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MQTT vs. Kafka: An IoT Advocate's Perspective

influxdata.com

Post reply on HN
Print view View original

11–20 of 122 posts

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#11

Post by avereveard »

mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

> advertising the impossible

eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#12

Post by alecthomas »

mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

> I'm a big fan of advertising the impossible on the front page.

Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#13

Post by codeflo »

alecthomas wrote: post #12
mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

> I'm a big fan of advertising the impossible on the front page. Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...

The impossibility of "exactly once" is a theorem, not an opinion.

Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter.

We all know that. It's called "at least once".

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#15

Post by [deleted] »

mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

[deleted]

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#16

Post by rad_gruchalski »

Jemaclus wrote: post #2

This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed. I'd be interested in a comparison that is actually apples-to-apples instead of introd…

Comparing mqtt to kafka is already apples vs oranges. Adding the schema registry to this is like throwing a pitaya into the mix.

Edit: after reading the article a couple of times it’s clear this isn’t a comparison. “Vs” in the title is the problem. The first impression would have been better if the title was something like ”Kafka and MQTT“.

To be honest, as kafka and mqtt often reside next to each other, they complement each other. A better use case would have been an app combining both: show when should mqtt hand over to kafka.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#17

Post by simplotek »

codeflo wrote: post #13
alecthomas wrote: post #12

Earlier quoted context omitted.

> I'm a big fan of advertising the impossible on the front page. Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...

The impossibility of "exactly once" is a theorem, not an opinion. Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idemp…

> The impossibility of "exactly once" is a theorem, not an opinion.

It's quite likely that your definition of what "exactly once" means differs from the one followed by MQTT. As this issue was documented years ago, I doubt this is a relevant argument to have, unless we want to feel smart by criticising others.

https://www.eejournal.com/2015/05/28/is-exactly-once-deliver...

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#18

Post by [deleted] »

mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

[deleted]

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#19

Post by Scubabear68 »

justinclift wrote: post #3

An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.

For shops light on DevOps-fu, Confluent hosted Kafka is popular for just this reason.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#20

Post by ryanjshaw »

Jemaclus wrote: post #2

This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed. I'd be interested in a comparison that is actually apples-to-apples instead of introd…

The author is also incorrect about message keys - these are optional and you would only use them for strict ordering when using > 1 partition, or log compaction where "latest is greatest" is good enough.
Post reply on HN