Why was Apache Kafka created?

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

Re: Why was Apache Kafka created?

#6

Post by clippy99 » Sat, Aug 23, 2025, 7:25 PM UTC

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Re: Why was Apache Kafka created?

#7

Post by atombender » Sat, Aug 23, 2025, 7:27 PM UTC

isaacremuant wrote: post #4

polynomial wrote: post #3

Why was it named that is also a question.

> Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work. From Wikipedia.

A system optimized for writing could also describe the machine in Kafka's "In the Penal Colony".

Re: Why was Apache Kafka created?

#8

Post by clippy99 » Sat, Aug 23, 2025, 7:28 PM UTC

isaacremuant wrote: post #4

polynomial wrote: post #3

Why was it named that is also a question.

> Jay Kreps chose to name the software after the author Franz Kafka because it is "a system optimized for writing", and he liked Kafka's work. From Wikipedia.

Kafkaesque to configure and get running for simple tasks.

Re: Why was Apache Kafka created?

#9

Post by zug_zug » Sat, Aug 23, 2025, 7:28 PM UTC

My only complaint with this article is that it seems to be implying kafka that linkedIn's problem couldn't have been solved with a bunch of off-the-shelf tools.

Re: Why was Apache Kafka created?

#10

Post by oulipo2 » Sat, Aug 23, 2025, 7:29 PM UTC

clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Have you tried Redpanda?