151–160 of 229 posts
> 17 PB/day Are they really generating data at this scale? I cant even imagine a system which creates and stores this much data this fast.
Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)
Earlier quoted context omitted.
I thought Kafka ditched the zookeeper
I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode 1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...
https://www.confluent.io/blog/zookeeper-to-kraft-with-conflu...
https://www.infoq.com/news/2025/06/linkedin-northguard-xinfr...
> According to LinkedIn's engineers, Kafka had become increasingly difficult to manage at LinkedIn's scale (32T records/day, 17 PB/day, 400K topics, 150 clusters).
Wtf is LinkedIn doing that they create 17 PB/day?????
> 17 PB/day Are they really generating data at this scale? I cant even imagine a system which creates and stores this much data this fast.
Where does this 17 PB/day number come from? I didn't quote any numbers directly. Looking at the 2012 paper, it implies a 1.35TB per day (they store 9.5TB across all topics at 7d retention)
https://www.linkedin.com/blog/engineering/infrastructure/int...
Earlier quoted context omitted.
> Why they'd need Kafka was always our first question, never got a good answer from a single one of them "To follow the hype train, Bro" is often the real answer. > If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. Or just freaking MQTT. MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messag…
> Or just freaking MQTT. Disclaimer: I'm a dev and I'm not very familiar with the actual maintenance of kafka clusters. But we run the aws managed service version (MSK), and it seems to just pretty much work. We send terrabytes of data through kafka asynchronously, because of its HA properties and persistent log, allowing consumers to consume in their own time and put the data where it needs to be. So imagine, many a…
MQTT is a publish/subscribe protocol for large scale distributed messaging, often used in small embedded devices or factories. It is made for efficient transfer of small, often byte sized payloads of IoT device data. It does not replace Kafka or RabbitMQ - messages should be read off of the MQTT broker as quickly as possible. ( I know this from experience - MQTT brokers get bogged down rapidly if there are too many messages "in flight")
A very common pattern is to use MQTT for communications, and then Kafka or RabbitMq for large scale queuing of those messages for downstream applications.
Earlier quoted context omitted.
I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…
> Why they'd need Kafka was always our first question, never got a good answer from a single one of them "To follow the hype train, Bro" is often the real answer. > If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. Or just freaking MQTT. MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messag…
Kafka isn’t a queue. It’s overkill to use it as one.
Kafka is a great place to persist data for minutes, hours or days before it’s processed. It fully decouples producers and consumers. It’s also stupidly complex and very hard to operate reliably in an HA configuration.
MQTT is good for when data needs to leave or enter your cloud, but persistence is bolted on (at least it is in mosquitto), so a crash means lost data even though you got a PUBACK.
It was later open-sourced in 2011 and became a top-level project under the Apache Software Foundation in 2012.
The creators went on to co-found Confluent, a company that provides commercial support and enterprise features around Kafka.
This might be hyperbolic, but I think Kafka (or at least the concept of event driven architecture for sharing data across many systems) is one of the most under-rated technologies. It's used at a lot of big corps but is never talked about.
Distributed ledgers like Bitcoin do store transitions as events, and that's because nodes need the transitions to valid the next state. So you might say that Bitcoin is a widely run piece of software, using an event driven architecture.
Not every system needs to have all of it's state transitions available for for efficient reading. And often times you can derive the state transition from the previous and next state if you really need them (Git does that). Even though Git can compute all of the state transitions for the system, it doesn't store events, it stores snapshots.
It was created to teach me the concept of love-hate relationships
I wanted to write a comment on this topic, but after several tries this thread is where I ended up because it describes my sentiment as well. The arguments in the article are very compelling. But as soon as you choose Kafka you realize the things you hate. Many of the reasons are stupid things - like it uncovers otherwise unimportant bugs in your client code. Or that it just makes experimenting a hassle because it en…
I don’t remember where I read this, but someone made the observation that writing a stream processing system is about 3x harder than writing a batch system, exactly for all the reasons you mentioned. I’m looking at replacing some of our Kafka usage with a clickhouse table that’s ordered and partitioned by insertion time, because if I want to do stuff with that data stream, at least I can do a damn SQL query.
Why was it named that is also a question.
> Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work. From Wikipedia.