Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

131–140 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#131

Post by splix »

gilbetron wrote: post #119
ashtonkem wrote: post #108

Earlier quoted context omitted.

I’ve deployed Java apps for years, the idea that I must “admit” to Java being painful to deploy in order to be taken seriously is laughable. Like any other tool, Java has its pros and its cons, but you’re being hyperbolic.

Ok, you have a java app I want on my system, what are the steps I need to do? (I'm on Ubuntu 18.04)

I guess you mean that Java 11 is not available on 18.04 by default, right? Java 11 was release in September 2018, so I guess Ubuntu 18.04, which came in April 2018, didn't have any chance to include it. Therefore you have to make an extra step to setup a ppa repo with software that didn't exist at the moment of release.

Otherwise, at least with Ubuntu 19.04 which I have, it's simple `sudo apt install openjdk-11-jdk-headless`

Re: Kafka Removing Zookeeper Dependency

#132

Post by apta »

gilbetron wrote: post #119
ashtonkem wrote: post #108

Earlier quoted context omitted.

I’ve deployed Java apps for years, the idea that I must “admit” to Java being painful to deploy in order to be taken seriously is laughable. Like any other tool, Java has its pros and its cons, but you’re being hyperbolic.

Ok, you have a java app I want on my system, what are the steps I need to do? (I'm on Ubuntu 18.04)

You can ship the JVM with your app such that it's self-contained and system independent. Apps like IntelliJ already do this. And now with jlink, you can strip it down to make deployment sizes even smaller by only shipping the parts of the JVM that your app uses.

Re: Kafka Removing Zookeeper Dependency

#133

Post by splix »

Tsarbomb wrote: post #114
femiagbabiaka wrote: post #82

Earlier quoted context omitted.

It is a good faith argument — it’s literally the whole point. Native binaries bypass all of that.

Good luck when you need to deploy to a system with a different version of libc.

Also with different version of openssl

Re: Kafka Removing Zookeeper Dependency

#134

Post by apta »

axaxs wrote: post #67
karmakaze wrote: post #63

Earlier quoted context omitted.

To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.

This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.

You can ship the JVM with your app such that it's self-contained and system independent. Apps like IntelliJ already do this. And now with jlink, you can strip it down to make deployment sizes even smaller by only shipping the parts of the JVM that your app uses.

Re: Kafka Removing Zookeeper Dependency

#135

Post by tamale »

kitotik wrote: post #120
elric wrote: post #88

Earlier quoted context omitted.

Seriously? When was the last time you deployed something as a standalone, static binary? Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Just about everything requires some kind of runtime, some kind of dependencies or some kind of environment. I can go "dnf install firefox" just as easily as I can go "dnf install ant". Both of these come with a boatload of dependenc…

> Hardly anything with any amount of complexity is shipped like that The entire HashiCorp stack and Cockroachdb comes to mind. I’m sure there are things that are much more complex in the world, but they do some fairly heavy lifting. > especially not cross-platform I find it drastically simpler to download a single binary for each platform I need to deploy to(macOS,Linux,FreeBSD) than it is to get a consistent Java en…

You're kinda omitting the insane amount of configuration you can give to pretty much every hashicorp product and cockroachdb

I think the big difference is that Java projects TEND to be more 'up front' about their configuration options, with shipped default files with a lot of the options already set to some default value, while projects like the ones you mentioned require you to look up every property yourself and set it to something if you want to change it from the default.

Re: Kafka Removing Zookeeper Dependency

#136

Post by agallego »

apta wrote: post #128
agallego wrote: post #66

Earlier quoted context omitted.

p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1 ~/nohup1.txt &

It's known that golang optimizes for latency at the expense of throughput, and it doesn't give you the option to change this if your requirements change. This is the power of the JVM. In any case, the JVM now ships with ZGC, a low latency GC, and it may be worth running the benchmark with it. There's also another low latency GC in the works called Shenandoah.

We wrote it in c++ on top of seastar.io

Re: Kafka Removing Zookeeper Dependency

#137

Post by StreamBright »

koffiezet wrote: post #92
StreamBright wrote: post #76

Earlier quoted context omitted.

Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?

Just the fact that you have to ask all these things proves the point?

Which one is easier? Adding a few lines of GC configuration or rewriting a service in C++ using Seastar and binding processes to cores?

What sort of performance measurement uses default configurations? What is even the point of not tuning the GC to your application workload?

I have spent the last 15 years on running Java apps in production and to optimize for the p99 latency is really not that hard. Optimize p99.99999 is a whole different subject though. I don't understand what is the point of comparing a default GC setting that is for hello world applications to a software that is optimized every way possible. Apples to oranges. It is a grat marketing gimmick though. Look ma no performance! Look here, so much faster. We live in a single dimension word, yay!

Re: Kafka Removing Zookeeper Dependency

#138

Post by manigandham »

StreamBright wrote: post #137
koffiezet wrote: post #92

Earlier quoted context omitted.

Just the fact that you have to ask all these things proves the point?

Which one is easier? Adding a few lines of GC configuration or rewriting a service in C++ using Seastar and binding processes to cores? What sort of performance measurement uses default configurations? What is even the point of not tuning the GC to your application workload? I have spent the last 15 years on running Java apps in production and to optimize for the p99 latency is really not that hard. Optimize p99.9999…

Easier for who? At some point, the architecture needs to change to overcome a fundamental limit. Developers take on that work to make the performance and operations easier for their users.

Seastar is the foundation of Scylla, which shows that rewriting in C++ can deliver magnitudes more performance which is not possible by just tuning Cassandra on the JVM. In fact, Datastax has now copied the Scylla approach in Cassandra but still lags behind drastically with performance.

Re: Kafka Removing Zookeeper Dependency

#139

Post by ashtonkem »

gilbetron wrote: post #119
ashtonkem wrote: post #108

Earlier quoted context omitted.

I’ve deployed Java apps for years, the idea that I must “admit” to Java being painful to deploy in order to be taken seriously is laughable. Like any other tool, Java has its pros and its cons, but you’re being hyperbolic.

Ok, you have a java app I want on my system, what are the steps I need to do? (I'm on Ubuntu 18.04)

wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/publi... | sudo apt-key add -

sudo add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/

sudo apt-get install adoptopenjdk--hotspot

Yes, it’s a few extra steps, but it’s not exactly Sisyphean now is it? Given the complexity of modern CI/CD pipelines, and the trend towards continuous deployment, making sure that a JVM is deployed alongside the uberjar is a pretty small ask. Would a single binary be easier? Sure. Would I change languages just for that? No, other factors are more important IMHO.

Re: Kafka Removing Zookeeper Dependency

#140

Post by lostapathy »

apta wrote: post #123
buzzkillington wrote: post #116

Earlier quoted context omitted.

>Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Including the JVM. The paths alone.

You can ship the JVM with your code (uber jar), so no paths to set up. And now with jlink, you can even strip it down to only parts of the JVM that your code uses, to optimize size.

From the perspective of a java developer, this makes total sense.

From the perspective of a sysadmin who just wants to deploy an app that ships as a jar and doesn't come pre-packaged with any convenience features like you mentioned, it's an awful lot to work out and learn.

Post reply on HN