Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

71–80 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#71

Post by pjmlp »

agallego wrote: post #66
pjmlp wrote: post #61

Earlier quoted context omitted.

How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1 ~/nohup1.txt &

Thanks.

Re: Kafka Removing Zookeeper Dependency

#72

Post by nine_k »

tbrock wrote: post #16
closeparen wrote: post #14

Earlier quoted context omitted.

Keystores are clunky, but other than that, why exactly?

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

Normally you just run your one JVM app on your VM. The real balance is basically between the heap size and page cache size. You might want to tweak it, as much as the GC algorithm and other memory details. This applies if you understand them.

If you don't, you just start the JVM with defaults, and it configures itself for optimal performance, using its idea of optimal.

Re: Kafka Removing Zookeeper Dependency

#73

Post by tiew9Vii »

silisili wrote: post #59
elric wrote: post #48

Earlier quoted context omitted.

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

They are really simple if done correctly. You have a single fat jar so can do java -jar myapp.jar which is almost as easy as your static binary but prefixed with two extra words.

It’s also trivially easy to package as a docker image.

What makes things like Zookeeper and Kafka to deploy is they are complex applications. You can write them in any other language and the deployment will still remain as non trivial.

Re: Kafka Removing Zookeeper Dependency

#74

Post by weego »

axaxs wrote: post #67
karmakaze wrote: post #63

Earlier quoted context omitted.

To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.

This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.

That's not really a good faith argument. That's the equivalent of saying that a major problem with NodeJS is you have to pick which version to install before you deploy.

I deploy single package jvm files most days having not had to consider the deploy environment for a about 4 years.

Re: Kafka Removing Zookeeper Dependency

#75

Post by sz4kerto »

chillacy wrote: post #69
sz4kerto wrote: post #60

Earlier quoted context omitted.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection, and then adjusted our heap size / code accordingly. It works well.

If you need really hard limits on collection then that's a tricky problem, but that's also tricky when you're managing memory yourself.

Re: Kafka Removing Zookeeper Dependency

#76

Post by StreamBright »

agallego wrote: post #66
pjmlp wrote: post #61

Earlier quoted context omitted.

How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1 ~/nohup1.txt &

Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?

Re: Kafka Removing Zookeeper Dependency

#77

Post by tinco »

DiabloD3 wrote: post #56
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

All you're doing is exchanging one runtime, the JVM, that happens to be well understood, highly optimized, and has a decade long track record, and supported by every major company on Earth... for Go's runtime, one that isn't well understood, does not have a track record, and is a product of a company that is known to just abandon popular services for no apparent reason. You also seem to have misunderstood what causes…

What about the JVM is well understood? The most common trope about it is that it's supposed to be the fastest VM ever because, but Java devs always manage to build slow apps in it. So much for understanding it well.

The claim that Go does not have a track record is ridiculous. It's whole premise is that it's made to power essential parts of the largest IT operation in the world.

And there's nothing for Google to cancel, it's open source and they even made the effort of translating it to Go so it's super easy to work on.

And I'm not fanboying over Go, haven't written a Go service in years, but denying Go is an excellent platform to build these kinds of apps on is ridiculous.

Re: Kafka Removing Zookeeper Dependency

#78

Post by StreamBright »

snidane wrote: post #36
agallego wrote: post #21

Earlier quoted context omitted.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because most people using JVM in production understand that your claim is without merit and performance in the JVM land can be tuned to the workload and finally it is never a single dimension decision to chose a language/ runtime. JVM had many additional properties that make it an excellent choice for big data use cases.

Re: Kafka Removing Zookeeper Dependency

#79

Post by StreamBright »

agallego wrote: post #62
sz4kerto wrote: post #60

Earlier quoted context omitted.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

The JVM is fast. (though I'd say the best thing about the JVM is the amazingly large test suite it maintains). I think he was alluding to the fact it is much easier to develop predictable systems in a language that forces you to deal with the constraints up front. - i.e.: writing your own memory allocator for custom pools knowing exactly the latency, throughput, assembly generated for it. Not to mention that a lot of…

True. I think the rejection of operating systems managing resources the way it is today (dynamic resource binding) is coming. Pinned threads make a ton of sense in HPC.

Re: Kafka Removing Zookeeper Dependency

#80

Post by StreamBright »

chillacy wrote: post #69
sz4kerto wrote: post #60

Earlier quoted context omitted.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

GC without tuning. With optimization, not so much.
Post reply on HN