Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

121–130 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#121

Post by devonkim »

silisili wrote: post #59
elric wrote: post #48

Earlier quoted context omitted.

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

It depends upon what you’re looking at in a deployment. Static binaries are the gold standard of “simple” in theory for ops folks but are not architecture neutral by definition. The JVM JAR spec is pretty well known as well and is not a surprise with plenty of documentation or blog posts to help out. Quirks do show up from crossing boundaries into native land like DNS and sockets behaviors, but a basic JAR does give some advantages over a random native binary in a tarball when it comes to dependency management which is tougher with a static binary. It’s not obvious which version of a library is compiled into a static binary at first glance compared to unzipping and checksumming the JARs in a package.

There’s definitely some laborious parts of the Java packaging setup (the whole resources, meta-inf / manifest setup reeks of YAGNI problems) but similar to Go’s lack of expressiveness being a feature, the Java ecosystem is fundamentally designed for organizations that separate developers from the systems where the software is run, and this is either helpful or hurtful entirely depending upon the organization’s needs.

I’ve never really had a problem deploying anything based upon the packaging - it’s a minor part compared to various obscure configuration files, ConfigMaps, or environment variable injections that bother me more, and that has nothing to do with a package or even language in itself.

Re: Kafka Removing Zookeeper Dependency

#122

Post by jonathanoliver »

koffiezet wrote: post #93
jonathanoliver wrote: post #39

Earlier quoted context omitted.

I'm an active Go user and I've been working almost exclusively in Go since 2011. I have a feeling that over time, the number of knobs on Go will increase much like it has in Java. I'm trying to think back to my first experiences with Java in 1995. It was so simple compared to the overhead of getting things working in C/C++. So Java's got like 25+ years on Go and I would speculate that the number of knobs on Go in 25…

Go is already 10yo, so I suppose this will be in 15 years?

Oops. Ya, 15. Off by 10 error!

Re: Kafka Removing Zookeeper Dependency

#123

Post by apta »

buzzkillington wrote: post #116
elric wrote: post #88

Earlier quoted context omitted.

Seriously? When was the last time you deployed something as a standalone, static binary? Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Just about everything requires some kind of runtime, some kind of dependencies or some kind of environment. I can go "dnf install firefox" just as easily as I can go "dnf install ant". Both of these come with a boatload of dependenc…

>Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Including the JVM. The paths alone.

You can ship the JVM with your code (uber jar), so no paths to set up. And now with jlink, you can even strip it down to only parts of the JVM that your code uses, to optimize size.

Re: Kafka Removing Zookeeper Dependency

#124

Post by please-reread »

jonathanoliver wrote: post #33
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Someone is adding journaling to the NATS protocol to make it Kakfa like: https://liftbridge.io/ It was just released as v1.0 a few weeks ago.

> make it Kakfa like

Kakfa?

Re: Kafka Removing Zookeeper Dependency

#125

Post by apta »

elric wrote: post #90
michaelt wrote: post #81

Earlier quoted context omitted.

I use Java for a living. Whether you think Java is very easy or a pain in the ass depends on what you're doing. Consider the fact Ubuntu 18.04's default repos come with a pretty ancient version of Java. And if you want a newer version, you have to use some third-party source. And the fact there are both OpenJDK and Oracle JDKs to choose from. Will that be JDK or JRE? Headless? Depending on the combination of JDK and…

I don't know which JVMs are or aren't in Ubuntu. But Ubuntu 18 is over 2 years old. Almost every major linux distribution has stable, LTS versions of Java 8 and 11 -- which are the currently supported LTS versions in general. A package can easily add a dependency on a specific Java version. And unlike many other tools, if you have multiple versions of Java, all you usually need to do is add the right one to the $PATH…

You can also ship the JVM directly with your product, that way it's system independent. Even better with jlink.

Re: Kafka Removing Zookeeper Dependency

#126

Post by apta »

gen220 wrote: post #102
pjmlp wrote: post #96

Earlier quoted context omitted.

Not really, after all most native code compilers have endless amount of configuration options as well, while Go still falls behind many use cases. Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box. All of which comes back to the original point that many times isn't the language, rather how i…

Go’s mission statement isn’t to be faster than Java or to replace java. Its to (1) compile faster (2) compile into single binaries with no dynamic links (3) natively support concurrency and parallelism with M:N routines:threads. Go will never replace Java for Android because (1) it doesn’t use a VM, so it would need to compile for every arch that android runs on (2) it would require a bug-compatible port of Android.…

> (1) compile faster

This isn't really true for non-trivial code bases. And even if it is, it's not a big difference in practice, especially with incremental compilation where I find it actually much quicker to change a couple of files and re-run unit tests compared to golang which has to spit out a multi-dozen MB binary each time, taking 5-6+ seconds.

> (2) compile into single binaries with no dynamic links

Java is getting AOT compilation which will do the same.

> (3) natively support concurrency and parallelism with M:N routines:threads.

Java is getting those as well: http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...

Re: Kafka Removing Zookeeper Dependency

#127

Post by skywhopper »

elric wrote: post #48
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

Wow, this does not match my experience at all. The JVM itself is harder to install than any native-binary service I've ever used. And that's before you get around to running something like Kafka or Zookeeper on top of it. Maybe "not all things" are a pain to deploy, but everything I've ever run on it or looked into running has been.

I'm particularly curious about how you deal with TLS in the JVM without keystores? I'd love to hear more about that.

Also, I have to laugh about the heap settings not "technically" being required. Sure, but how many JVM-based services have you actually run in production without setting them? If it's more than zero, then I congratulate you on your luck.

Re: Kafka Removing Zookeeper Dependency

#128

Post by apta »

agallego wrote: post #66
pjmlp wrote: post #61

Earlier quoted context omitted.

How 10x faster? Benchmarks available? If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?

p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1 ~/nohup1.txt &

It's known that golang optimizes for latency at the expense of throughput, and it doesn't give you the option to change this if your requirements change. This is the power of the JVM. In any case, the JVM now ships with ZGC, a low latency GC, and it may be worth running the benchmark with it. There's also another low latency GC in the works called Shenandoah.

Re: Kafka Removing Zookeeper Dependency

#129

Post by apta »

lllr_finger wrote: post #103
sz4kerto wrote: post #75

Earlier quoted context omitted.

Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection,…

Once you start talking hundreds-to-thousands requests/second, 500ms is an incredibly long time and you're well past simple tweaks to GC. Tuning GC to a high degree is non-deterministic black magic, which is not what you're looking for at that point. Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at…

The JVM now ships with ZGC, a low latency GC targetted at exactly that. https://wiki.openjdk.java.net/display/zgc/Main

Re: Kafka Removing Zookeeper Dependency

#130

Post by karmakaze »

axaxs wrote: post #67
karmakaze wrote: post #63

Earlier quoted context omitted.

To run a Java jar-with-dependencies you only need to `java -jar app-with-deps.jar` which isn't really that hard at all.

This skips the steps of choosing and installing a JVM, which is a bit daunting. My distro gives like 4 different versions. And worse, some applications only run on certain versions. By default, there is no 'java' command.

This is a question you have to answer once for your project. Similar to choosing which OS, database, library, or framework to use. There's no avoiding having knowledge or learning when choosing dependencies.

Just about any of the implentations will work fine. You only have to concern yourself with avoiding use of the Oracle runtime in production without a license. If in doubt, install the latest AdoptOpenJDK[0] Hotspot version that's compatible with your app.

[0] https://adoptopenjdk.net/

Post reply on HN