Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

91–100 of 229 posts

Re: Why was Apache Kafka created?

#91

Post by mdaniel »

AtlasBarfed wrote: post #58
rvz wrote: post #35

Earlier quoted context omitted.

> Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play. Kafka is used because the Java folks don't want to learn something new due to job security, even though there are faster and compatible alternatives that exist today. Rather use Redpanda, than continue to use Kafka and then complain about how resource intensive it…

I thought Kafka ditched the zookeeper

I wish; KIP-500[1] was the "let's use native Raft" implementation but I have never once in my life seen anyone using Kafka in KIP-500 mode

1: https://cwiki.apache.org/confluence/display/kafka/kip-500:+r...

Re: Why was Apache Kafka created?

#92

Post by vips7L »

vlovich123 wrote: post #60
bdangubic wrote: post #22

Earlier quoted context omitted.

you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :) jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory

Technically kind of true but at the same time Android apps are predominantly Java/Kotlin. It speaks more to Java just having a bad desktop story. But it’s also why Android devices need 2x the ram

That has nothing to do with Java. The Android runtime is NOT Java/OpenJdk.

Re: Why was Apache Kafka created?

#93

Post by chatmasta »

sea-gold wrote: post #32
varbhat wrote: post #18

Does anyone use https://nats.io here? I have heard good things about it. I would love to hear about the comparisons between nats.io and kafka

There is a good comparison between NATS, Kakfa, and others here: https://docs.nats.io/nats-concepts/overview/compare-nats

It’s on the NATS website and “nats” appears in the URL three times, so maybe this isn’t the most objective source.

Re: Why was Apache Kafka created?

#94

Post by laughing_man »

atomicnumber3 wrote: post #51
adev_ wrote: post #27

Earlier quoted context omitted.

> Why they'd need Kafka was always our first question, never got a good answer from a single one of them "To follow the hype train, Bro" is often the real answer. > If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. Or just freaking MQTT. MQTT has been battle-proven for 25 years, is simple and does perfectly the job if you do not ship GBs of blobs through your messag…

It's resume-driven development. It honestly can make sense for both company and employee. Companies get standard tech stacks people are happy to work with, because working with them gets people experience with tech stacks that are standard at many companies. It's a virtuous cycle. And sure even if you need just a specific thing, it's often better to go slightly overkill for something that's got millions of stack over…

That's what happened where I worked. The people making the tech decisions were worried they weren't "keeping up" and committed us all-in on kafka. That decision cost the company millions.

Re: Why was Apache Kafka created?

#95

Post by chatmasta »

pluto_modadic wrote: post #85

lots of people not considering UNPHAT - https://gist.github.com/rponte/67c78e6b3ee349a6e14cb8fb155b7... / https://stevenschwenke.de/pragmaticSoftwareEngineeringUNPHAT - some solutions didn't exist in the past. you evaluate your landscape and your problem, and try to find a good fit.

Kafka was created at LinkedIn when it was already quite a large web platform, to solve the problem of distributing a unique newsfeed to a wide audience… so while they weren’t Google, they were still closer to Google than the targets of the “UNPHAT” expression. You can apply UNPHAT to most of the people using Kafka today, but it’s not fair to apply it to LinkedIn when they created it.

Re: Why was Apache Kafka created?

#96

Post by chamomeal »

throwaway8242 wrote: post #75
buckle8017 wrote: post #28

Earlier quoted context omitted.

Java the language and Java the runtime are fine. The way most Java code is written is terrible Enterprise factory factory factory.

That doesn't match my experience in the last 15 years working for 3 companies (one was a big enterprise, one medium sized and one startup) Maybe I have been lucky, or that the practice is more common in certain countries or eco systems? Java has been a very productive language for me, and the code has been far from the forced pattern usage that I have read horror stories about.

Have you gotten to use loom/virtual threads? I’ve heard pretty interesting stuff about em, but haven’t really spent the time to get into it yet. It’s pretty exciting and tbh gives me an easy elevator pitch to JVM world for people outside of it

Re: Why was Apache Kafka created?

#97

Post by mldbk »

slipperydippery wrote: post #16
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Whatever efficiency may hypothetically be possible with Java, you can in-fact spot a real world Java program in the wild by looking for the thing taking up 10x the memory it seems like it should need… when idle. Yes yes I’m sure there are exceptions somewhere but I’ve been reading Java fans using benchmarks to try to convince me that I can’t tell which programs on my computer are Java just by looking for the weirdly…

I held the same view as you when I was 22, more than 15 years ago.

With over 15 years of professional experience since then, my perspective has shifted: Java demonstrates its strength when stability, performance, and scalability are required (e.g. bloody enterprise)

A common misconception comes from superficial benchmarking. Many focus solely on memory consumption, which often provides a distorted picture of actual system efficiency.

I can point to EU-scale platforms that have reliably served over 100 million users for more than a decade without significant issues. The bottleneck is rarely the language itself, it is the depth of the team’s experience.

Re: Why was Apache Kafka created?

#98

Post by vlovich123 »

vips7L wrote: post #92
vlovich123 wrote: post #60

Earlier quoted context omitted.

Technically kind of true but at the same time Android apps are predominantly Java/Kotlin. It speaks more to Java just having a bad desktop story. But it’s also why Android devices need 2x the ram

That has nothing to do with Java. The Android runtime is NOT Java/OpenJdk.

No but it does speak to the memory overhead of tracing GC vs ref counting as garbage collection strategies.

Re: Why was Apache Kafka created?

#99

Post by hvb2 »

selkin wrote: post #84
mrweasel wrote: post #24

Earlier quoted context omitted.

I previously helped clients setup and run Kafka clusters. Why they'd need Kafka was always our first question, never got a good answer from a single one of them. That's not to say that Kafka isn't useful, it is, in the right setting, but that settings is never "I need a queue". If you need a queue, great, go get RabbitMQ, ZMQ, Redis, SQS, named pipes, pretty anything but Kafka. It's not that Kafka can't do it, but yo…

KIP-932[0] adds queue semantics for Kafka. You may still want to use another queue though: as always, no one size fits all. [0] https://cwiki.apache.org/confluence/display/KAFKA/KIP-932%3A...

I CAN drive a Ferrari to the grocery store.

Sure, it can do it. But it's not efficient or what it's good at.

Re: Why was Apache Kafka created?

#100

Post by [deleted] »

dxxvi wrote: post #80
erulabs wrote: post #67

Kafka's ability to ingest the firehose and present it as a throttle-able consumable to many different applications is great. If you're thinking "just use a database", it's worth noting that SQL databases are _not well suited_ to drinking from a firehose of writes, and that distributed SQL in 2012 was not a thing. Kafka was one of the first systems that fully embraced the dropping of the C from CAP theorem, which was…

> turn to Redis Cluster Streams instead. So much simpler to manage and so much cheaper to run I don't have any experience with Redis Cluster Streams. Could you please tell us how it is simpler to manage? IMO, installing and managing a Kafka cluster in a non Fortune 100 scale is simple enough: run 1 java command for zookeeper, run another java command for a broker (with recent version of Kafka, zookeeper is not needed…

[deleted]
Post reply on HN