Live data from Hacker News

MQTT vs. Kafka: An IoT Advocate's Perspective

influxdata.com

111–120 of 122 posts

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#111

One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means yo…

Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#112

An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.

To be fair, it's not like it's solving a trivial problem. High throughput, reliable and highly available message queuing is just hard.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#113

One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means yo…

Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..

I think you are overcomplicating the problem for no reason.

You build your system from simple guarantees:

* message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations.

* the object on S3 is immutable. This means it does not matter when you consume it, it stays the same.

* the message on Kafka is immutable. This means it does not matter when you consume it, it stays the same.

When the client reads the message off of Kafka topic, it just downloads the additional payload from S3. The payload is guaranteed to be there and exactly the same content as published. Once the message is fully processed, it commits this to Kafka topic and that's done. If the processing fails, the processing will be retried later by this or another node. The payload is still there until somebody decides to delete it.

Deletion can be done in many different ways. You could have metadata for all those objects (Kafka topic is your metadata database!) and see what is the oldest timestamp on the offsets on all partitions still not committed. Then you delete from S3 all objects that are older that that. This requires that you publish to Kafka in the same order as you publish to S3 (within each partition).

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#114

Earlier quoted context omitted.

Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..

I think you are overcomplicating the problem for no reason. You build your system from simple guarantees: * message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations. * the object on S3 is immutable. This means it does not matter when you consume it…

Yes I understood your proposed solution (and have no particular issues with it). I was specifically asking how you went about deleting things (i.e. the last paragraph of your response). Did you actually implement it that way in the end?

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#115

Earlier quoted context omitted.

Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?

If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, th…

Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka.

Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#116

Earlier quoted context omitted.

Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image. What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?

Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker…

Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#117

Earlier quoted context omitted.

Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker…

Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.

Definitely. Focus on solving the business problem first, grow the business. When you have the need to make a switch, you will find a way forward by rebuilding your your MQTT infrastructure to fit your scale. If you ever need to.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#118
Since nobody else mentioned it: Often you'll want to have multiple consumers of a given topic for load-balancing/failover - that's on the consumer side. The support for this in the MQTT standard is poorly defined, and popular libraries handle it poorly to the point that I can't recommend trying it. Message queues right-and-proper load-balance consumers by locking the entire topic once per delivery; Kafka does it by assigning "partitions" of a topic to different consumers, and thus you have faster, lock-free delivery (Kafka is essentially a "hot-rodded" message queue, just like hot-rodding a car by removing certain "proper" guard rails).

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#119
post #96

I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt). So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micr…

Having been down this road, I strongly recommend you lock down your requirements before drowning in the sea of transport options. There are a lot of offerings out there that, from the device perspective, basically amount to a reinvention of HTTP(S). Many of these systems exist with the intent to retain the underlying TCP (really, the expensive-to-establish TLS) stream, which is not as much of a concern after HTTP 1.2 retained existing connections.

What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter.

If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races.

Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#120
post #107
post #92

Earlier quoted context omitted.

> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost. What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT A…

> What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered? ZeroMQ is blazing fast, damn simple and battlefield proven. However it is also pretty low level. Things like ACL, topic filtering or user management often need to be reimplemented on top of it. That is not the case for most MQTT blockers. Also if security is a a big concern for you, I would pick up MQTT+HAPro…

[deleted]
Post reply on HN