81–90 of 122 posts
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
Earlier quoted context omitted.
Maybe something AMQP related instead?
Architecturally AMQP is just bigger brother of MQTT, to the point some queue brokers (like RabbitMQ) can accept both. MQTT 5 made difference smaller too.
What I liked about a mix of three of those was that I could connect Azure Event Hub with RabbitMQ and ActiveMQ, they would all speak to each other via Qpid.
Earlier quoted context omitted.
> Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? When the use case warrants it and when downstream consumers need to be notified of a change in the data or its state. That is, in reactive architectures. A data streaming platform (of which Kafka is one example) will push the data, whereas a database (relational or not) requires to be explicitly po…
> A data streaming platform (of which Kafka is one example) will push the data
The Kafka broker keeps hot data in RAM. The clients pulls: https://github.com/apache/kafka/blob/3.3.2/clients/src/main/.... Most often it’s a frequent pull with a short timeout.
This is one of the major differences from MQTT. MQTT broker pushes to the subscriber when the latter is connected, or stores for redelivery if subscriber is offline (qos 1 and 2) but the redelivery will also be a real push on reconnect.
I was a fan of NATS and Kafka in the past, but AWS tooling makes IoT relatively easy.
Cheers
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
Check out http://redpanda.com for a simpler to manage, and 10x faster Kafka
We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated.
Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo.
What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic.
Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3. But it's a key property of such a system. I'm guessing Amazon's IoT gateway does this, because once you've thought about it hard enough it becomes obvious this is how it needs to work.
1: https://www.influxdata.com/blog/mqtt-vs-kafka-iot-advocates-...
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
Check out http://redpanda.com for a simpler to manage, and 10x faster Kafka
Good article (along with parts 2 and 3). Are there key differences in secure networking constructs (TLS, mTLS, VPN, whitelisted IPs, open ports, etc.) in the options described: + inbound to Kafka clusters and Kafka Connect? + inbound to Mosquitto MQTT broker? + inbound to Telegraf? + inbound to influxDB?
For the others, secure them using the same tech you use to secure your database. Ideally they wouldn't be accessible from the internet at all.
I need the ability to queue data on the device to deal with patchy connectivity and some policy for deleting messages in the queue. For example, I might throw away unsent location updates that are "stale" to save space. I might need to prioritize some messages, e.g. "lithium-ion battery pack overheating".
I may be running on embedded hardware that is too small to run Linux. The connection to the modem might be serial.
Data size and bandwidth usage can make a difference. I might get 2MB/month for $3. Bytes count if I want to send frequent updates to get more precision on the location.
I generally know exactly what messages I am sending, so using a compiled format like gRPC or COAP (https://en.wikipedia.org/wiki/Constrained_Application_Protoc...) can be better than JSON.
I may need to get through multiple layers of network address translation, making it different to send messages back. So keeping a persistent TCP connection can help. But using TCP for one-off connections wastes bytes, so UDP can be useful if I don't care about losing packets.
I may want to encrypt or digitally sign your messages.
So, I end up doing a lot of work to handle queueing locally, but using a pretty simple message-oriented binary protocol to send to the server. The server can do whatever it wants, e.g. write it to a Kafka queue.
Amazon's IoT framework nails a lot of these points. https://aws.amazon.com/blogs/compute/building-an-aws-iot-cor...