Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

MQTT vs. Kafka: An IoT Advocate's Perspective

influxdata.com

Post reply on HN
Print view View original

51–60 of 122 posts

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#51

Post by speedgoose »

I hear from everyone using Kafka in production that it is hell unless you use Confluent.

I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances.

I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is that retained messages are not sent to wildcard subscribers), but it should work with AMQP.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#52

Post by Gwypaas »

jck wrote: post #48
septune wrote: post #38

Earlier quoted context omitted.

Forget MQTT, Redis as a PUB/SUB will do 99% of the job most of the time.

Redis 7 supports sharded pubsub. Also, redis streams are excellent and perform really well for Kafka lite type use cases.

Until you need backpressure, then you are left with awful out-of-band hacks and hope for the best.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#53

Post by gvtek0 »

adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

>However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train

Don't look now but this is how people end up with k8s as well. "We need Kubernetes because we need containers." Google et al convinced people it's the only way to run containers in prod.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#54

Post by outworlder »

justinclift wrote: post #3

An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.

> In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches.

Compared to what?

I have the opposite experience. For example, ingesting large amounts of log data. Kafka could handle an order of magnitude more events compared to Elasticsearch. Even if the data ultimately ended up in ES, being able to ingest with Kafka improved things considerably. We ended up getting an out of the box solution that does just that (Humio, now known as LogScale).

Similar experience when replacing RabbitMQ with Kafka. None "just works" and there's always growing pains in high throughput applications, but that comes with the territory.

Is Kafka the source of headaches, or is it Zookeeper? Usually it's Zookeeper for me (although, again, Zookeeper has difficult problems to solve, which is why software packages use ZK in the first place).

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#55

Post by EdwardDiego »

speedgoose wrote: post #51

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is t…

Welp, here's a dissenting opinion - it's not.

I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi).

It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#56

Post by EdwardDiego »

adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

> notoriously hard to operate (Hello ZooKeeper)

Eh? ZooKeeper is rock solid.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#57

Post by outworlder »

FridgeSeal wrote: post #22
hkt wrote: post #9

A better comparison with Kafka is redis streams. Similar semantics, a fraction of the operational overhead.

Provided your surrounding tools plug into Redis streams. Oh and provided you don’t need the ordering and parallelism guarantees of Kafka’s partitions. Oh and provided you don’t need the same level of durability and fault tolerance, so yeah, exactly the same.

Exactly. Tradeoffs exist everywhere, in both directions.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#58

Post by EdwardDiego »

victor106 wrote: post #30
adev_ wrote: post #25

The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…

>Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions). 100% this. Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db using skip locked as a queue mechanism and the dev teams productivity tripled and our total cost of ownership decreased dramatically. Don’t think t…

Yeah, if you're able to replace Kafka with PG then your stack didn't need Kafka, so that's good call.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#59

Post by amath »

speedgoose wrote: post #51

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is t…

Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.

Re: MQTT vs. Kafka: An IoT Advocate's Perspective

#60

Post by speedgoose »

amath wrote: post #59
speedgoose wrote: post #51

I hear from everyone using Kafka in production that it is hell unless you use Confluent. I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances. I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is t…

Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.

I haven’t took the time but I should. Redpanda is written in C++ and I tend to prefer safer programming languages, redis being the exception.
Post reply on HN