Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

101–110 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#101

Post by lllr_finger »

sz4kerto wrote: post #60
snidane wrote: post #36

Earlier quoted context omitted.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

LMAX also spent a ton of time doing non-standard things with data structures to optimize cache and prevent heap allocation: https://martinfowler.com/articles/lmax.html

Unnecessary allocations will be your bottle neck at some point when shutting data around.

Re: Kafka Removing Zookeeper Dependency

#102

Post by gen220 »

pjmlp wrote: post #96
koffiezet wrote: post #92

Earlier quoted context omitted.

Just the fact that you have to ask all these things proves the point?

Not really, after all most native code compilers have endless amount of configuration options as well, while Go still falls behind many use cases. Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box. All of which comes back to the original point that many times isn't the language, rather how i…

Go’s mission statement isn’t to be faster than Java or to replace java. Its to (1) compile faster (2) compile into single binaries with no dynamic links (3) natively support concurrency and parallelism with M:N routines:threads.

Go will never replace Java for Android because (1) it doesn’t use a VM, so it would need to compile for every arch that android runs on (2) it would require a bug-compatible port of Android. Because Kotlin runs on the JVM and can run on top of Java codebases, it doesn’t have to leap over these hurdles.

There are other reasons you can come up with I’m sure, but those are the biggest that stick out to me. Notably it has nothing to do with “which language is better”.

They optimize for different use cases, and that’s OK. That being said, if you were writing Android from scratch and were only targeting ARM (an equally silly comparison meant to highlight the differences in the languages), you’d be hard pressed to justify Java over Go.

Re: Kafka Removing Zookeeper Dependency

#103

Post by lllr_finger »

sz4kerto wrote: post #75
chillacy wrote: post #69

Earlier quoted context omitted.

It's fast if you measure throughput or median latency but it tends to have pretty monstrous worst case latencies due to GC.

Depends. Java's GC can be tuned an infinitum. It's not a simple task and requires knowledge, but that's the beauty of it: if you need low-ish latencies, then you can tune the GC to target that instead of throughput. For example we're using relatively large heaps (partly due to inefficiencies in our code), but we still want to stay under 500 msec/request or so. So we told the G1 GC to target 150 msecs per collection,…

Once you start talking hundreds-to-thousands requests/second, 500ms is an incredibly long time and you're well past simple tweaks to GC. Tuning GC to a high degree is non-deterministic black magic, which is not what you're looking for at that point.

Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at a FAANG - so it isn't exactly an uncommon scenario for developers.

Re: Kafka Removing Zookeeper Dependency

#104

Post by gen220 »

skyde wrote: post #53
lchengify wrote: post #42

I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong. Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project.…

i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state. It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling resta…

It might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper’s source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd.

Also, etcd powers many critical open source projects, so there are many institutional eyes that actively contribute to its improvement. IME if we ever encountered an issue at work with ZK, we found it impossible to trace it down to a bug that we could fix and upstream. Etcd’s been easier in this regard.

Re: Kafka Removing Zookeeper Dependency

#105

Post by snidane »

sz4kerto wrote: post #60
snidane wrote: post #36

Earlier quoted context omitted.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM. Look at stuff like LMAX. Java can be lightning fast.

You might be correct - it may be only my ignorant experience interacting with Apache Big Data technologies of present day and wondering what the hell they are doing under the hood to take them so long to process simple stuff in bulk.

Perhaps open source engineers are not "paid" enough to spend countless hours carefully optimizing their programs on JVM. Properly paid engineers working on proprietary technology can surely bend and twist JVM and make it perform. It might be possible, but costly due to complexity and unpredictable nature of it.

It reminds me of world of SQL where you have to run your query with many small modifications and hope that the optimizer will generate a sensible query plan while the query is still readable. That's the cost of building on top unpredictable systems. You wonder why you can't simply get access to the lower level - physical plans - and program on that level directly, since you know what your desired outcome on that level is anyway.

I still wonder why new projects for performant data processing are not written for example in Rust since for this task it has all the upsides and none of the downsides of JVM.

Re: Kafka Removing Zookeeper Dependency

#106

Post by gen220 »

KenanSulayman wrote: post #94

Finally! Having had to deal with properly setting zookeeper up on multiple mesos clusters was so painful and recovering from outages via s3 backups so excruciating.. I feel somewhat ashamed that in two out of five cases we simply went with the managed Kafka by AWS because the team didn’t feel confident enough to maintain the cluster.

Honestly this is probably the right thing to do. My cynical conspiracy is that the ZK dep was organized by cloud vendors to keep customers coming :)

It’s been at least a year and a half since we’ve had severe data loss on our homebrew Kafka cluster, but after the first couple you never look at Kafka+Zk the same way... both iirc were due to leadership election bugs that had been reported many months ago with no progress on a solution.

I have no idea how AWS internally puts up with this. I wouldn’t be surprised if they’d replaced the ZK dependency internally years ago.

Re: Kafka Removing Zookeeper Dependency

#107

Post by rahilb »

tbrock wrote: post #99
vips7L wrote: post #30

Earlier quoted context omitted.

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

Why isn’t that the default? Unless I say otherwise, use it. Imagine launching every app and having to say ok you get 300mb but not more!

It’s the default because Java has a history, and it was set when everyone was deploying war files to machines with 512mb total memory and the default worked for 95% of people for the past 25 years. In fact, it’s still good enough for the chocolate factory that has brainwashed you into drinking the static binary milkshake.

Of course there are ways to use all the available ram when you deploy your kubernetes pods, you just have to consult the documentation.

Honestly this criticism is ridiculous I can’t believe I even bothered to reply.

Re: Kafka Removing Zookeeper Dependency

#108

Post by ashtonkem »

silisili wrote: post #59
elric wrote: post #48

Earlier quoted context omitted.

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

I’ve deployed Java apps for years, the idea that I must “admit” to Java being painful to deploy in order to be taken seriously is laughable.

Like any other tool, Java has its pros and its cons, but you’re being hyperbolic.

Re: Kafka Removing Zookeeper Dependency

#109

Post by growse »

tbrock wrote: post #97
staticassertion wrote: post #22

Earlier quoted context omitted.

But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step. That's the power of having tunables.

Then they chose better defaults because I never have to think about it.

Just because the defaults are better for you doesn't mean they're better for everyone.

Re: Kafka Removing Zookeeper Dependency

#110

Post by lima »

gen220 wrote: post #104
skyde wrote: post #53

Earlier quoted context omitted.

i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd. in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state. It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling resta…

It might sound kind of trite, but it has less to do with the replication algorithm and more to do with the fact that Zookeeper’s source is quite complicated vs, say etcd, so there are more opportunities for subtle bugs to appear. I encourage you to look at the bug section of the change log for zookeeper, and also the feature list of ZK vs etcd. Also, etcd powers many critical open source projects, so there are many i…

Agreed, etcd is rock solid.

With regards to Kafka, it's probably easier and more robust to add their own consensus layer rather than switching to etcd - Kafka is already a distributed system built by a team of distributed systems engineers. It makes sense for them to build their own consensus, deeply integrated with the replication mechanism, rather than relying on an external database.

Post reply on HN