Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

61–70 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#61

Post by pjmlp »

agallego wrote: post #21
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

How 10x faster? Benchmarks available?

If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

Re: Kafka Removing Zookeeper Dependency

#62

Post by agallego »

sz4kerto wrote: post #60
snidane wrote: post #36

Earlier quoted context omitted.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it.

Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work.

In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.

Re: Kafka Removing Zookeeper Dependency

#63

Post by karmakaze »

silisili wrote: post #59
elric wrote: post #48

Earlier quoted context omitted.

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.

Re: Kafka Removing Zookeeper Dependency

#64

Post by bananabreakfast »

DiabloD3 wrote: post #56
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

All you're doing is exchanging one runtime, the JVM, that happens to be well understood, highly optimized, and has a decade long track record, and supported by every major company on Earth... for Go's runtime, one that isn't well understood, does not have a track record, and is a product of a company that is known to just abandon popular services for no apparent reason. You also seem to have misunderstood what causes…

Complaining about Google sunsetting consumer products as a means to disparage their technical contributions to open source software is incredibly trite and a very tired meme.

So, so much of Google depends on Go. It is extremely well understood, has an excellent track record and has zero chance of being "abandoned"

Re: Kafka Removing Zookeeper Dependency

#65

Post by vips7L »

thayne wrote: post #45
vips7L wrote: post #30

Earlier quoted context omitted.

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

That is a terrible idea. The JVM basically never returns RAM to the OS, and in my experience, if you run a java process long enough, the max heap size ends up being the used heap size.

G1 the default allocator returns heap to the os only when you set -G1PeriodicGCInterval and both ZGC and Shenandoah do it automatically.

Re: Kafka Removing Zookeeper Dependency

#66

Post by agallego »

pjmlp wrote: post #61
agallego wrote: post #21

Earlier quoted context omitted.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

p99 latency.

No startup times of course. This is for pushing a simple 2 petabyte workload.

No AOT compilers, just download kafka bin distribution 2.4.1

Just launching 6 or 7 of these.

  bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \
    --record-size 1024 \
    --topic sfo \
    --num-records $((1 ~/nohup1.txt &

Re: Kafka Removing Zookeeper Dependency

#67

Post by axaxs »

karmakaze wrote: post #63
silisili wrote: post #59

Earlier quoted context omitted.

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.

This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.

Re: Kafka Removing Zookeeper Dependency

#69

Post by chillacy »

sz4kerto wrote: post #60
snidane wrote: post #36

Earlier quoted context omitted.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

Re: Kafka Removing Zookeeper Dependency

#70

Post by agallego »

chillacy wrote: post #69
sz4kerto wrote: post #60

Earlier quoted context omitted.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

+1K on this. tail latencies are bad.
Post reply on HN