Why was it named that is also a question.
> Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work. From Wikipedia.
[deleted]
Why was Apache Kafka created?
11–20 of 229 posts
Re: Why was Apache Kafka created?
#12Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
If you are using this sort of redis queue (https://redis.io/glossary/redis-queue/) with PUSH/POP vs fan-out you're working on a very different sort of problem than what Kafka is built for.
Like the article says, fan-out is a key design characteristic. There are "redis streams" now but they didn't exist back then. The durability story and cluster stories aren't as good either, I believe, so they can probably take you so far but won't be as generally suitable depending on where your system goes in the future. There are also things like RedPanda that speak Kafka w/o the Java.
However, if you CAN run on a single node w/o worrying about partitioning, you should do that as long as you can get away with it. Once you add multiple partitions ordering becomes hard to reason about and while there are things like message keys to address that, they have limitations and can lead to hotspotting and scaling bottlenecks.
But the push/pop based systems also aren't going to give you at-least-once guarantees (looks like Redis at least has a "pop+push" thing to move to a DIFFERENT list that a single consumer would manage but that seems like it gets hairy for scaling out even a little bit...).
Re: Why was Apache Kafka created?
#13Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
> and it feels bloated (Java!)
I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Re: Why was Apache Kafka created?
#14My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.
What off the shelf tools in 2012 would you propose, exactly?
Re: Why was Apache Kafka created?
#15I worked on this while I was at LI and I think the major selling point back then was Replayability of messages but it was something similar to what you would get with Pub/Sub. We could also have multiple clients listening and processing same messages for their own purposes so you could use the same queue and have different clients process them as they wanted.
Re: Why was Apache Kafka created?
#16Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.
> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle.
Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly slow ones, when I in fact very much could, for 25ish years.
Java programs have a feel and it’s “stuttery resource hog”. Whatever may be possible with the platform, that’s the real-world experience.
Re: Why was Apache Kafka created?
#17My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.
And what would that off-the-shelf software have been?
Re: Why was Apache Kafka created?
#18Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka
Re: Why was Apache Kafka created?
#19It was created to teach me the concept of love-hate relationships
Re: Why was Apache Kafka created?
#20Earlier quoted context omitted.
> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?
Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…
The JVM eats a chunk of memory in order to make its garbage collector more efficient. Think of it like Linux's page cache.
I haven't worked with too much Java, but I suspect that the distaste many have for it is due to its wide adoption by large organizations and the obfuscating "dressed up" tendency of the coding idioms used in large organizations.
The runtime isn't inherently slow, but maybe it's easier to write slow programs in Java.