21–30 of 122 posts
+ inbound to Kafka clusters and Kafka Connect?
+ inbound to Mosquitto MQTT broker?
+ inbound to Telegraf?
+ inbound to influxDB?
A better comparison with Kafka is redis streams. Similar semantics, a fraction of the operational overhead.
Oh and provided you don’t need the ordering and parallelism guarantees of Kafka’s partitions.
Oh and provided you don’t need the same level of durability and fault tolerance, so yeah, exactly the same.
I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it. But I did see: This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once I'm a big fan of advertising the impossible on the front page.
> I'm a big fan of advertising the impossible on the front page. Do you mean like Confluent do? https://www.confluent.io/blog/exactly-once-semantics-are-pos...
An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
From a technology perspective it's been rock solid for years in my experience.
Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the effects of partitioning, resetting offsets on a live topic, aggressive retention policies etc.
However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster.
And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost.
Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions).
If all you need is a simple stupid publish/subscribe broker with topics/auth management, do a favour to yourself, stay away from it.
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received.
How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?
An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
What issues did you run into? From a technology perspective it's been rock solid for years in my experience. Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the…
I.e. an event occurs which causes an order of magnitude more messages than usual to be produced for a couple of hours, and because ingest and processing flows are out of whack, a backlog forms. Management wants things back in sync ASAP, and so green lights increasing the partition count on the topic, usually doubling it.
In an event driven architecture that is fairly well tuned for normal traffic this can have the same downstream effect, and those topics up their partition counts as well in response.
Once anomalous traffic subsides, teams go to turn down the now over-partitioned topics only to learn that that was a one way operation and now they’re stuck with that many partitions, and the associated cost overhead.
Also if I see another team try to implement “retries” or delayed processing on messages by doing some weird multi-topic trickery I’m going to lose my mind. Kafka is a message queue, not a job queue, and not nearly enough engineers seem to grok that.
An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
An important question not mentioned in this article - and may not have been known by the author - is how much (Dev)Ops burden do each of these add? In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches. That's in fairly high throughput environments though, no idea if it "just works" flawlessly in easy going ones.
If I had a need for Kafka in my current role, I’d probably give Confluent and Red Panda offerings a shot.
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema. However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster. And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server…
100% this.
Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db using skip locked as a queue mechanism and the dev teams productivity tripled and our total cost of ownership decreased dramatically.
Don’t think twice, think 10 times if you really need Kafka