One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means yo…
MQTT vs. Kafka: An IoT Advocate's Perspective
111–120 of 122 posts
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#112An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#113One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means yo…
Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..
You build your system from simple guarantees:
* message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations.
* the object on S3 is immutable. This means it does not matter when you consume it, it stays the same.
* the message on Kafka is immutable. This means it does not matter when you consume it, it stays the same.
When the client reads the message off of Kafka topic, it just downloads the additional payload from S3. The payload is guaranteed to be there and exactly the same content as published. Once the message is fully processed, it commits this to Kafka topic and that's done. If the processing fails, the processing will be retried later by this or another node. The payload is still there until somebody decides to delete it.
Deletion can be done in many different ways. You could have metadata for all those objects (Kafka topic is your metadata database!) and see what is the oldest timestamp on the offsets on all partitions still not committed. Then you delete from S3 all objects that are older that that. This requires that you publish to Kafka in the same order as you publish to S3 (within each partition).
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#114Earlier quoted context omitted.
Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..
I think you are overcomplicating the problem for no reason. You build your system from simple guarantees: * message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations. * the object on S3 is immutable. This means it does not matter when you consume it…
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#115Earlier quoted context omitted.
Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?
If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, th…
Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#116Earlier quoted context omitted.
Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image. What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?
Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker…
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#117Earlier quoted context omitted.
Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker…
Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#118Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#119I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt). So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micr…
What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter.
If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races.
Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#120Earlier quoted context omitted.
> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT A…
> What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? ZeroMQ is blazing fast, damn simple and battlefield proven. However it is also pretty low level. Things like ACL, topic filtering or user management often need to be reimplemented on top of it. That is not the case for most MQTT blockers. Also if security is a a big concern for you, I would pick up MQTT+HAPro…