Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

41–50 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#41

Post by colin_mccabe »

nealabq wrote: post #10

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.

But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.

Unfortunately, this is not correct. BookKeeper stores a lot of information in ZookKeeper. By extension, Pulsar (which is based on BookKeeper) also stores a lot of metadata there as well.

For example, from the BK documentation ( https://zookeeper.apache.org/doc/r3.3.6/bookkeeperOverview.h... ):

An application first creates a ledger before writing to bookies through a local BookKeeper client instance. Upon creating a ledger, a BookKeeper client writes metadata about the ledger to ZooKeeper. Each ledger currently has a single writer. This writer has to execute a close ledger operation before any other client can read from it. If the writer of a ledger does not close a ledger properly because, for example, it has crashed before having the opportunity of closing the ledger, then the next client that tries to open a ledger executes a procedure to recover it. As closing a ledger consists essentially of writing the last entry written to a ledger to ZooKeeper, the recovery procedure simply finds the last entry written correctly and writes it to ZooKeeper.

Re: Kafka Removing Zookeeper Dependency

#42

Post by lchengify »

I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong.

Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project. But in practice, I've never had a smooth experience getting Zookeeper to run consistently, especially with Kafka.

For sure, part of it is that we treat it like oxygen, in that if it's gone for a few seconds everything just dies. But having dealt with similar systems both proprietary and open source, my opinion is that Zookeeper just hasn't risen to the challenges of its users in the past 5 years. If the next generation of software architects want to use open source streaming or distributed systems, Zookeeper needs to be rewritten or removed.

Also shout out to the confluent.io team: I never paid for your enterprise license, but without your blog posts, docker images, or slack room, I never would have been able to get Kafka working. Thanks again!

Re: Kafka Removing Zookeeper Dependency

#43

Post by simtel20 »

vips7L wrote: post #30
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

If you haven't done -Xmx= before, you may not be aware that you're giving a bum steer here.

Re: Kafka Removing Zookeeper Dependency

#44

Post by Twirrim »

tbrock wrote: post #16
closeparen wrote: post #14

Earlier quoted context omitted.

Keystores are clunky, but other than that, why exactly?

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's a black box optimizer for databases, which is designed to take the user away from having to find and tweak all the hundreds of possible tunables in the database, https://github.com/cmu-db/ottertune / https://ottertune.cs.cmu.edu/

I keep wanting the same for JVM. It's probably nice that there are so many ways to adjust the JVM to best suit a very wide range of workloads, but from a practical perspective I rarely have spare time to go diving through the guts to experiment. Let me point something at a one-box, and it can tweak parameters to its hearts content, so I can export and apply to the rest of the fleet.

Re: Kafka Removing Zookeeper Dependency

#45

Post by thayne »

vips7L wrote: post #30
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

That is a terrible idea. The JVM basically never returns RAM to the OS, and in my experience, if you run a java process long enough, the max heap size ends up being the used heap size.

Re: Kafka Removing Zookeeper Dependency

#46

Post by skyde »

agallego wrote: post #21
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

so this use raft for topic data not just metadata ?

Re: Kafka Removing Zookeeper Dependency

#47

Post by skyde »

tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

are you using Jocko in production?

Re: Kafka Removing Zookeeper Dependency

#48

Post by elric »

tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question.

You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them shifts some of the complexity of dealing with certificates from the developer to the system administrator. It's an implementation choice.

The only other complaint of yours I could find ITT was about setting min/max heap size... not only is it extremely convenient to be able to do that, it also hasn't ever technically been required, and the defaults have been Good Enough for most uses since Java 5 came out in 2004. The JVM will figure it out for you, and if it's too conservative or too aggressive for you, you can tune it.

If Kafka is a PITA to deploy, fine, maybe that's fair criticism. Not all things on the JVM are pain to deploy.

Re: Kafka Removing Zookeeper Dependency

#49

Post by skyde »

saurik wrote: post #12
sagichmal wrote: post #8

Earlier quoted context omitted.

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

100 % agree we need good (etch/zookeeper) as a library.

I wrote something similar but that provide only (lock/lease) using paxos.

The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 node should be in the quorum ...

Re: Kafka Removing Zookeeper Dependency

#50

Post by skyde »

manigandham wrote: post #20
hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

This is the difference between a library and a framework/project. For example, there's a great Raft library for Go [1] that any project can use to implement distributed consensus without a separate running program. I find this to be a better approach with the same collective development and community testing advantages but without requiring more operational overhead. 1. https://github.com/hashicorp/raft

this lib u is great and 1000 time cleaner better tested and easier to use compared to trying to reuse etcd raft implementation :-)
Post reply on HN