Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

91–100 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#91

Post by elric »

PopeDotNinja wrote: post #87
michaelt wrote: post #81

Earlier quoted context omitted.

I use Java for a living. Whether you think Java is very easy or a pain in the ass depends on what you're doing. Consider the fact Ubuntu 18.04's default repos come with a pretty ancient version of Java. And if you want a newer version, you have to use some third-party source. And the fact there are both OpenJDK and Oracle JDKs to choose from. Will that be JDK or JRE? Headless? Depending on the combination of JDK and…

+1 for this. A headless install of Java is not straight forward. Doable? Yes. Pain in the butt? Yes.

yum install java-latest-openjdk-headless

There, you're done. Need fonts? Install fonts. Pain in the butt? Where?

Re: Kafka Removing Zookeeper Dependency

#92

Post by koffiezet »

StreamBright wrote: post #76
agallego wrote: post #66

Earlier quoted context omitted.

p99 latency. No startup times of course. This is for pushing a simple 2 petabyte workload. No AOT compilers, just download kafka bin distribution 2.4.1 Just launching 6 or 7 of these. bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \ --record-size 1024 \ --topic sfo \ --num-records $((1 ~/nohup1.txt &

Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?

Just the fact that you have to ask all these things proves the point?

Re: Kafka Removing Zookeeper Dependency

#93

Post by koffiezet »

jonathanoliver wrote: post #39
tptacek wrote: post #25

Earlier quoted context omitted.

They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.

I'm an active Go user and I've been working almost exclusively in Go since 2011. I have a feeling that over time, the number of knobs on Go will increase much like it has in Java. I'm trying to think back to my first experiences with Java in 1995. It was so simple compared to the overhead of getting things working in C/C++. So Java's got like 25+ years on Go and I would speculate that the number of knobs on Go in 25…

Go is already 10yo, so I suppose this will be in 15 years?

Re: Kafka Removing Zookeeper Dependency

#94

Post by KenanSulayman »

Finally! Having had to deal with properly setting zookeeper up on multiple mesos clusters was so painful and recovering from outages via s3 backups so excruciating.. I feel somewhat ashamed that in two out of five cases we simply went with the managed Kafka by AWS because the team didn’t feel confident enough to maintain the cluster.

Re: Kafka Removing Zookeeper Dependency

#95

Post by bsaul »

elric wrote: post #88
silisili wrote: post #59

Earlier quoted context omitted.

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

Seriously? When was the last time you deployed something as a standalone, static binary? Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Just about everything requires some kind of runtime, some kind of dependencies or some kind of environment. I can go "dnf install firefox" just as easily as I can go "dnf install ant". Both of these come with a boatload of dependenc…

I'm not sure what you mean. Deploying static binaries was the main way to go before, and is now the standard approach with go. Pretty much every service developed in go is deployed as a standalone statically compiled binary, and i would assume it's the same for code developped in Rust.

Re: Kafka Removing Zookeeper Dependency

#96

Post by pjmlp »

koffiezet wrote: post #92
StreamBright wrote: post #76

Earlier quoted context omitted.

Did you try to tune GC (as most Kafka production systems would have GC configured)? Which JVM version?

Just the fact that you have to ask all these things proves the point?

Not really, after all most native code compilers have endless amount of configuration options as well, while Go still falls behind many use cases.

Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box.

All of which comes back to the original point that many times isn't the language, rather how it is written and what tools one makes use of.

If Go is so much better than Java, Google would have replaced it already on Android with Go (battery life and such), instead they went with a mix of AOT/JIT with PGO, introducing Kotlin, while gomobile efforts were never given any serious consideration not even for the NDK.

Re: Kafka Removing Zookeeper Dependency

#97

Post by tbrock »

staticassertion wrote: post #22
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.

Then they chose better defaults because I never have to think about it.

Re: Kafka Removing Zookeeper Dependency

#98

Post by bsaul »

elric wrote: post #91
PopeDotNinja wrote: post #87

Earlier quoted context omitted.

+1 for this. A headless install of Java is not straight forward. Doable? Yes. Pain in the butt? Yes.

yum install java-latest-openjdk-headless There, you're done. Need fonts? Install fonts. Pain in the butt? Where?

And now you have a separate intermediate runtime layer between your code and the OS, with its own set of versions, patches, regression, bugs, backward compatibilities, etc. that may or may not have an impact on the code that you intend to deploy (or redeploy).

Statically compiled binaries are undoubtedly a plus regarding deployment.

Re: Kafka Removing Zookeeper Dependency

#99

Post by tbrock »

vips7L wrote: post #30
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

Why isn’t that the default? Unless I say otherwise, use it.

Imagine launching every app and having to say ok you get 300mb but not more!

Re: Kafka Removing Zookeeper Dependency

#100

Post by tbrock »

skyde wrote: post #47
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

are you using Jocko in production?

No, it didn’t support consumer groups last time I checked (but could!).
Post reply on HN