Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

21–30 of 229 posts

Re: Why was Apache Kafka created?

#21

Post by poemxo »

njitbew wrote: post #13
clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Starting up a Java program takes much longer than it should and that affects perception.

Re: Why was Apache Kafka created?

#22

Post by bdangubic »

slipperydippery wrote: post #16
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :)

jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory

Re: Why was Apache Kafka created?

#23

Post by jghn »

slipperydippery wrote: post #16
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

> taking up 10x the memory it seems like it should need… when idle.

The JVM tends to hold onto memory in order to make things faster when it does wind up needing that memory for actual stuff. However, how much it holds on to, how the GC is setup, etc are all tunable parameters. Further, if it's holding onto memory that's not being used, these are prime candidates to be stored in virtual memory which is effectively free.

Re: Why was Apache Kafka created?

#24

Post by mrweasel »

kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but you are making things harder than they needed to be.

Re: Why was Apache Kafka created?

#25

Post by cortesoft »

clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

It doesn't have to be 'hyper-scaled' to be needed, unless we have widely different definitions of hyper scale. Access logs from a few thousand servers with medium traffic will push you past any single instance service, and Kafka works great for that workload.

Re: Why was Apache Kafka created?

#26

Post by halfmatthalfcat »

poemxo wrote: post #21
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Starting up a Java program takes much longer than it should and that affects perception.

With AOT tho that should be somewhat moot.

Re: Why was Apache Kafka created?

#27

Post by adev_ »

mrweasel wrote: post #24
kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

> Why they'd need Kafka was always our first question, never got a good answer from a single one of them

"To follow the hype train, Bro" is often the real answer.

> If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka.

Or just freaking MQTT.

MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messaging system (which you should not do anyway).

Re: Why was Apache Kafka created?

#28

Post by buckle8017 »

njitbew wrote: post #13
clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Java the language and Java the runtime are fine.

The way most Java code is written is terrible Enterprise factory factory factory.

Re: Why was Apache Kafka created?

#29

Post by js4ever »

bdangubic wrote: post #22
slipperydippery wrote: post #16

Earlier quoted context omitted.

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :) jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory

[flagged]

Re: Why was Apache Kafka created?

#30

Post by nchmy »

varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

I dont have kafka experience, but nats is absolutely amazing. Just a complete pleasure to use, in every way.

https://www.synadia.com/blog/nats-and-kafka-compared

Post reply on HN