Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MQTT vs. Kafka: An IoT Advocate's Perspective

influxdata.com

Post reply on HN
Print view View original

101–110 of 122 posts

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#101

Post by heipei »

justinclift wrote: post #47
rad_gruchalski wrote: post #37

Earlier quoted context omitted.

Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…

Interesting. What are you thoughts on NSQ? https://github.com/nsqio/nsq Was looking at it earlier today, but haven't ever tried it out.

NSQ is a message broker that's intended to run in a distributed fashion, i.e. one broker per machine which is producing messages. Consumers then have to discover and connect to all brokers that carry messages on a given topic. NSQ persists messages to disk, but it does not have replication, so if one of the machines with the brokers dies (as in dead disk), all queued messages will be lost.

We use NSQ extensively and its a great fit for us. It's fast, lightweight, very easy to understand and work with, has great HTTP APIs for controlling the queues and publishing messages, etc. What it doesn't have is replication, so if a message must not be lost ever I'd reach for a distributed log, probably looking at Redpanda before Kafka, and possibly building my own on top of ScyllaDB since we're already running that.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#102

Post by sally_glance »

rad_gruchalski wrote: post #68
MuffinFlavored wrote: post #41

Earlier quoted context omitted.

> Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?

> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer…

Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image.

What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#103

Post by sally_glance »

physicles wrote: post #86

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system. We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka woul…

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#104

Post by jonquark »

rad_gruchalski wrote: post #71
jonquark wrote: post #45

Earlier quoted context omitted.

This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen

You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress. Example - you have 1 topic with 1 producer and 20M consumers. Each consumer i…

I agree there are cases where Kafka and MQTT are often used together. If you have lots of MQTT clients producing messages fanned-in to a small number of backend apps (or consuming a small number of wide fan out messages) people often combine Kafka as a fat pipe behind MQTT brokers (though there are alternatives, consuming messages from the brokers using e.g. MQTTv5 shared subs)

In more complicated situations (e.g. "outgoing" persistent messages buffered for individual clients (i.e. each client has a "message inbox"), Kafka is less obviously useful (it's an anti-pattern to try and random-seek the messages from Kafka topics as clients connect). In this kind of pattern, the main architecture pattern I see are clients partitioned across (highly available) MQTT brokers. If the messages come from MQTT clients directly to other clients (e.g. instant messaging (Facebook Messenger uses MQTT), having these broker in a cluster sharing a topic tree is very useful.

If the outgoing messages don't get buffered whilst the clients are off-line (e.g. because these are responses to client requests) then you don't need each client to be routed to a "home" broker that buffers the messages - it can connect anywhere.

It all depends on the shape of the message flow you're designing the system to support (but to say MQTT is a toy for small numbers of topics as it seemed to me that you argued up-thread seems misguided).

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#105

Post by FridgeSeal »

sally_glance wrote: post #103
physicles wrote: post #86

The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system. We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka woul…

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?

If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#106

Post by jinmingjian »

MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa.

But, sadly, people are often caught in their own loops, when there are or can be created better options.

I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/

This single binary data-service platform has proved:

1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level

With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka.

We have provided FREE full functionality community for testing: https://joinbase.io/products/

2. Use MQTT and Kafka together is unnecessary. This only makes your pipeline more complex, expensive, but much unstable and slower.

JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster.

3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered.

5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine: + sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker + basic SQL analytics: 3-4x faster than ClickHouse + HTTP interface concurrent queries: ~100x higher than ClickHouse + ... More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/

There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#107

Post by adev_ »

petre wrote: post #92
adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT A…

> What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered?

ZeroMQ is blazing fast, damn simple and battlefield proven. However it is also pretty low level.

Things like ACL, topic filtering or user management often need to be reimplemented on top of it. That is not the case for most MQTT blockers.

Also if security is a a big concern for you, I would pick up MQTT+HAProxy+TLS anytime over an ZMQ with CurveMQ and its home-made security.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#108

Post by adev_ »

rad_gruchalski wrote: post #71
jonquark wrote: post #45

Earlier quoted context omitted.

This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen

You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress. Example - you have 1 topic with 1 producer and 20M consumers. Each consumer i…

> So you need a fat pipe and some middle man [...] waiting subscribed to topics and waiting for messages.

Your use case is not Mr everybody use case nor the one presented in the article. Most usages of Kafka I have encountered in the wild is for notification delivery or telemetry report and of the order of few ~1000 msgs/s.

You do not need a fully distributed ordered log system for that. MQTT does the job for a fraction of the complexity and operational cost.

> . Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers,

Even at these scales, you can find some commercial MQTT brokers going over 20M msg/sec nowadays.

With OSS solutions, you could get your way there with some HAProxy + your favorite MQTT broker behind DNS load. balancing as long as you do not requires HA, scale only should not be the issue.

It would even play pretty nicely with anycast if you want to place your brokers at edge close to your customers and do some proper partitioning.

That is currently pretty much the case presented in the article. They just advertise telemetry report (very likely not HA) injected by a time series database.

Once again, if what you need is fully ordered distributed commit log for a complex scenario of event sourcing: Go for it, go for Kafka, it has been designed for that. But it is just not the case of most Kafka instances I see deployed in the wild, these ones are generally the result of quick Google-search driven engineering.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#109

Post by rad_gruchalski »

sally_glance wrote: post #102
rad_gruchalski wrote: post #68

Earlier quoted context omitted.

> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer…

Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image. What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?

Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker is where the problem starts. Imagine the following scenario:

You have 10 brokers, each broker handles 1M connections. You have a TopicA subscriber on broker 9 and 10, someone publishes a message to TopicA on broker 2. Your broker 2 needs to forward the message to brokers 9 and 10 but no others because if you do this for every message, you waste a ton of compute. So you need a method to distribute your connection table: which brokers currently have connections with subscriptions to TopicA. This is reasonably easy to embed in the broker for a relatively small number of brokers but if you need hundreds of brokers, your global connection table will not fit on each broker. Scale this up - millions of topics and millions of connected clients. You need a way to offload the connection table somewhere else, find a reliable method to distribute changes to your connection table. This is where we have successfully used Kafka - distributing the connection table and state changes.

Post reply on HN