Earlier quoted context omitted.
Check out http://redpanda.com for a simpler to manage, and 10x faster Kafka
Have you any experience with it in production?
Not yet! It's on my list. Have heard lots of good things from folks who are tho.
MQTT vs. Kafka: An IoT Advocate's Perspective
91–100 of 122 posts
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#92The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost.
What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered?
We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT API.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#93Earlier quoted context omitted.
Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…
You are absolutely right, but not every project needs reliable messaging. Many projects can be just fine with ephemeral messaging, they might already have some kind of managed scalable database and failures can often be handled on the client side. Often we insert messaging into an HTTP system, or replace an HTTP system. In those cases, the client is waiting for an acknowledgement from the system that sits on the othe…
> You are absolutely right, but not every project needs reliable messaging.
Until a manager comes on and decides it does and you have to redesign everything related to the message queue.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#94Earlier quoted context omitted.
Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…
> Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
> instead of storing rows in SQL with a timestamp
You realize a time series database is more appropriate for that?
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#95Earlier quoted context omitted.
What issues did you run into? From a technology perspective it's been rock solid for years in my experience. Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the…
Do you know of any places to learn those things? Kafka seems pretty interesting to me
- Confluent blog, particularly anything by Martin Kleppmann: https://www.confluent.io/blog/author/martin-kleppmann/
- "I Heart Logs" by Jay Krepps (free)
- "Kafka, The Definitive Guide" (also free)
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#96I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt).
So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micro-controller architecture, Raspberry Pi's, Arduino, PyBoard etc. via some sort of messaging service.
Clients connect to a page where they can "pub/sub" to scales to see the current weight of specific scales and using a remote device. The page acts as a UI to either manage inventory, register consumed ingredients, or register completed tasks.
So far most of the development scripts and test environments I've built all seem to do basically the same thing and will gather the data that's needed but I'm curios if anyone has dealt with anything like this in the wild and if there are any caveats that I'm missing or other technologies that would be better suited for what I'm trying to achieve?
Right now Kafka or RabbitMQ seem to be my main choices for message brokers mostly because they are fairly easy to setup via Docker. If anyone has any recommendation on libraries I should look into that would be awesome! The UI is coming together nicely, I started it in React but switched to Vue3 after I fell in love with the component architecture and composition API.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#97Controlling emergency generators via AWS is scary.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#98Earlier quoted context omitted.
Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two. Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message. Kafka is relatively small to medium n…
> Kafka is a distributed log When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed? Why do you need a sharded Kafka cluster? Most businesses are going to have Redis, SQL, and probably RabbitMQ. Where/why add Kafka to that stack?
If you can live with it (low traffic, small data set) you totally should. Where ‘low’ and ‘small’ are based on what a modern server can do, so they can go up quite a bit. And with shards and/or replicas you can get N times the throughput in an easy way
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#99This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed. I'd be interested in a comparison that is actually apples-to-apples instead of introd…
Comparing mqtt to kafka is already apples vs oranges. Adding the schema registry to this is like throwing a pitaya into the mix. Edit: after reading the article a couple of times it’s clear this isn’t a comparison. “Vs” in the title is the problem. The first impression would have been better if the title was something like ”Kafka and MQTT“. To be honest, as kafka and mqtt often reside next to each other, they complem…
Can confirm, I work with several workloads where MQTT is used to aggregate IoT data close to the source, which is then pushed into the cloud which uses Kafka.
Re: MQTT vs. Kafka: An IoT Advocate's Perspective
#100I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt). So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micr…
I once had to develop something similar. The solution my company had chosen was to transmit the weights and possible errors via a Websocket connection between the UI and the scale adapter - a small Java service doing the raw socket communication.
Weirdly the commands to stop weighing and set tare, get tare, etc. would be send via RabbitMQ from the UI server to the adapter instead of using the WS connection.