Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why was Apache Kafka created?

bigdata.2minutestreaming.com

Post reply on HN
Print view View original

41–50 of 229 posts

Re: Why was Apache Kafka created?

#41

Post by cameronh90 »

njitbew wrote: post #13
clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

> what exactly feels bloated about Java?

https://docs.spring.io/spring-framework/docs/2.5.x/javadoc-a...

Re: Why was Apache Kafka created?

#43

Post by closeparen »

clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Publishing an event to Kafka puts it “out there” in a way that guarantees it won’t be lost and allows any number of interested consumers, including the data warehouse, to deal with it at their leisure (subject to retention period which is typically like 72h). For us, your Kafka topics and their schemas are as much a part of your API as your gRPC IDLs. Something like Redis or 0MQ feels more appropriate for internal coordination between instances of the same service, or at least a producer that has a specific consumer in mind.

Re: Why was Apache Kafka created?

#44

Post by vlovich123 »

buckle8017 wrote: post #28
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Java the language and Java the runtime are fine. The way most Java code is written is terrible Enterprise factory factory factory.

But the perf is not reliable. If you want latency and throughput, idiomatic Rust will give you better properties. Interestingly even will Go for some reason has better latency guarantees I believe even though it’s GC is worse than Java.

Re: Why was Apache Kafka created?

#45

Post by p2detar »

js4ever wrote: post #29
bdangubic wrote: post #22

Earlier quoted context omitted.

you know why you don’t see many non-Java programs on your computer taking up 10x memory? because no one uses them to write anything :) jokes aside, we got a shift in the industry where many java programs were replaced by electron-like programs which now take 20x memory

[flagged]

Is this an AI-generated answer? Most of these are not even true, although I still would prefer Go for micro-services. I'll address just a bunch and to be clear - I'm not even a big Java fan.

- Quarkus with GraalVM compiles your Java app to native code. There is no JIT or warm up, memory footprint is also low. By the way, the JVM Hotspot JIT can actually make your Java app faster than your Go or Rust app in many cases [citation needed] exactly due to the hot path optimizations it does.

- GC tuning - I don't even know who does this. Maybe Netflix or some trading shops? Almost no one does this nowadays and with the new JVM ZGC [0] coming up, nobody would need to.

> You can’t ship a minimal standalone binary without pulling in a JVM.

- You'd need JRE actually, e.g., 27 MB .MSI for Windows. That's probably the easiest thing to install today and if you do this via your package manager, you also get regular security fixes. Build tools like Gradle generate a fully ready-to-execute directory structure for your app. If you got the JRE on your system, it will run.

> Dependency management and classpath conflicts historically plagued Java

The keyword here is "historically". Please try Maven or Gradle today and enjoy the modern dependency management. It just works. I won't delve into Java 9 modules, but it's been ages since I last saw a class path issue.

> J2EE

Is someone still using this? It is super easy writing a web app with Java+Javalin for example. The Java library and frameworks ecosystem is super rich.

> “Write once, run anywhere” costs: The abstraction layers that make Java portable also add runtime weight and overhead.

Like I wrote above, the HotSpot JIT is actually doing the heavy lifting for your in real time. These claims are baseless without pointing to what "overhead" is meant in practice.

---

0 - https://inside.java/2023/11/28/gen-zgc-explainer/ or https://www.youtube.com/watch?v=dSLe6G3_JmE

Re: Why was Apache Kafka created?

#46

Post by haddr »

clippy99 wrote: post #6

Startup founder here -- we tried it, and it feels bloated (Java!), bureaucratic and overcomplicated for what it is. Something like Redis queues or even ZMQ probably suffices for 90% of use cases. Maybe in hyper-scaled applications that need to be ultraperformant (e.g., realtime trading, massive streaming platforms) is where Kafka comes into play.

Couldn’t disagree more… if you go the ZMQ you are left alone handling many things you get in Kafka for free. If you have any sort of big data problems then good luck. You are going to reinvent the wheel.

Re: Why was Apache Kafka created?

#47

Post by von_lohengramm »

buckle8017 wrote: post #28
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

Java the language and Java the runtime are fine. The way most Java code is written is terrible Enterprise factory factory factory.

The problem is that writing genuinely performant Java code requires that you drop most if not all of the niceties of writing Java. At that point, why write Java at all? Just find some other language that targets the JVM. But then you're already treading such DIY and frictionful waters that just adopting some other cross-platform language/runtime isn't the worst idea.

Re: Why was Apache Kafka created?

#48

Post by dcminter »

cameronh90 wrote: post #41
njitbew wrote: post #13

Earlier quoted context omitted.

> and it feels bloated (Java!) I'm curious, what exactly feels bloated about Java? I don't feel like the Java language or runtime are particularly bloated, so I'm guessing you're referring to some practices/principles that you often see around Java software?

> what exactly feels bloated about Java? https://docs.spring.io/spring-framework/docs/2.5.x/javadoc-a...

Kafka does not use Spring.

Re: Why was Apache Kafka created?

#50

Post by AtlasBarfed »

kachapopopow wrote: post #5

As someone who has made the mistake of using kafka in a non enterprise space - it really seems like the etcd problem where you need more time to run etcd than to run whatever service you're providing.

If you're running a distributed system...

You're running a distributed system. They aren't simple.

Especially on AWS. AWS is really a double-bladed sword. Yeah, you'll get tutorials to set up whatever distributed system pretty quickly, but your nodes aren't nearly as reliable. Your networking isn't nearly as reliable. Your costs aren't nearly as reliable and administration. Headaches go up in the long run

Post reply on HN