Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MQTT vs. Kafka: An IoT Advocate's Perspective

influxdata.com

Post reply on HN
Print view View original

41–50 of 122 posts

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#41

Post by MuffinFlavored »

rad_gruchalski wrote: post #37
justinclift wrote: post #33

Earlier quoted context omitted.

Maybe something AMQP related instead?

Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…

> Kafka is a distributed log

When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed?

Why do you need a sharded Kafka cluster?

Most businesses are going to have Redis, SQL, and probably RabbitMQ.

Where/why add Kafka to that stack?

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#42

Post by twawaaay »

One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means you need less other code on your super restricted device.

That said, I don't understand all the complaining directed at Kafka in this thread. Kafka is a fantastic tool that provides unique properties and guarantees. As a tech lead/architect I love to have a good selection of tools for different situations. Kafka is very reliable tool that fils an important role of when creating distributed systems and is particularly nice because it is easy to reason about. The negative opinions I heard in the past are typically from people who try to use it for something that it is not well suited for (like efficient transfer of large volumes of data) or because they misunderstood how to use its guarantees to construct larger systems.

At one place I met a team who was completely lost with their overloaded Kafka instance and requested to get external help to "further scale and tune" it.

I just touched the piece of code on producer and on consumer to publish data in large files to S3 rather than push it all through Kafka. Instead, send a simple message to Kafka with the metadata and location of the payload in S3. And then the client to download it from the bucket. They were happy puppies in no time.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#43

Post by skrtskrt »

there’s a lot of “Kafka causes so many issues!” comments here.

I think it gets a bad rap because it gets introduced to orgs without the org having the requisite level of understanding. If your whole org is just on like a standard OLTP/OLAP setup, then suddenly there’s a Kafka queue, there’s going to he a serious learning curve and bumps along the way.

If you’re incorrectly putting async event brokers as the datastore where you should be putting a synchronous DB and then streaming from the DB to kafka with an outbox pattern, you’re going to have a bad time.

If you’re not modeling your queue depth and throughput you’re going to have a bad time.

If you’re not modeling your concurrency scenarios and synchronization, you’re going to have a bad time.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#44

Post by imglorp »

alexisread wrote: post #39
mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

With mqtt it depends on the broker, eg. Emitter.io can save them for a week etc. Offset for a client is usually stored on the broker, so if a client reconnects, all of the messages it has missed are forwarded to it. As mentioned in other answers, service levels have a defined meaning, which is different to the absolute theoretical meaning, and really is to do with the message aks. Really, as the article mentions, kaf…

nats +1

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#45

Post by jonquark »

rad_gruchalski wrote: post #32
MuffinFlavored wrote: post #26

Earlier quoted context omitted.

> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received. How many of us…

And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.

This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase

Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#46

Post by lmm »

adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

Kafka no longer requires zookeeper. If you need true master-master high availability from a datastore - which anyone who bothers with a load balancer for their application should demand, what's the point in running your application in a HA configuration if your datastore is a single point of failure - then to the best of my knowledge Kafka is still the least bad option available. It's not the easiest thing to operate, but I'll take it over Galera or Greenplum any day.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#47

Post by justinclift »

rad_gruchalski wrote: post #37
justinclift wrote: post #33

Earlier quoted context omitted.

Maybe something AMQP related instead?

Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…

Interesting. What are you thoughts on NSQ?

https://github.com/nsqio/nsq

Was looking at it earlier today, but haven't ever tried it out.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#48

Post by jck »

septune wrote: post #38
adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

Forget MQTT, Redis as a PUB/SUB will do 99% of the job most of the time.

Redis 7 supports sharded pubsub.

Also, redis streams are excellent and perform really well for Kafka lite type use cases.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#49

Post by itpragmatik »

adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

AWS SNS + SQS Fanout works pretty well and not too complicated.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#50

Post by jonquark »

mrkeen wrote: post #5

I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.

MQTT.org can't answer that as it's a web page for for a protocol. I've worked on platforms that do have a historian feature but it will vary from broker to broker.

(disclosure, I work on Eclipse Amlen and it does not - but people often rig it up to a subscriber that funnels (some/all) messages into databases

Post reply on HN