Kafka Removing Zookeeper Dependency

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story.

On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but it also means that more people are familiar with the quirks, limitations, requirements etc.... of these tools. For example, I think I would be frustrated to hear that K8s were implementing their own raft.

Re: Kafka Removing Zookeeper Dependency

#5

Post by eatonphil » Sat, May 16, 2020, 11:41 PM UTC

Here's the proposal in full.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A...

Re: Kafka Removing Zookeeper Dependency

#6

Post by smarterclayton » Sat, May 16, 2020, 11:42 PM UTC

hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

The difference is that Kubernetes isn’t a data store - we’d have to implement the full functionality of etcd. Kafka is a full featured data store that doesn’t need 80% of what zookeeper does.

This was actually an early complaint leveled against Kubernetes for things like proxying at the node level or implementing DNS. “Don’t reinvent the wheel!” Sometimes better administrative experiences exist only after a component absorbs some function previous systems expose.

Sometimes it’s better to own the parts of the problem that make your system simpler.

Re: Kafka Removing Zookeeper Dependency

#7

Post by justicezyx » Sun, May 17, 2020, 12:04 AM UTC

smarterclayton wrote: post #6

hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

The difference is that Kubernetes isn’t a data store - we’d have to implement the full functionality of etcd. Kafka is a full featured data store that doesn’t need 80% of what zookeeper does. This was actually an early complaint leveled against Kubernetes for things like proxying at the node level or implementing DNS. “Don’t reinvent the wheel!” Sometimes better administrative experiences exist only after a component…

No etcd don't need to be replicated for k8s API server.

Basically versioning + some form of fault tolerance is sufficient for k8s API server.

Re: Kafka Removing Zookeeper Dependency

#8

Post by sagichmal » Sun, May 17, 2020, 12:12 AM UTC

hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

Re: Kafka Removing Zookeeper Dependency

#9

Post by gregwebs » Sun, May 17, 2020, 12:13 AM UTC

hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

There are a lot of shared raft libraries now. It is much easier to share library code than a piece of a distributed system.

Re: Kafka Removing Zookeeper Dependency

#10

Post by nealabq » Sun, May 17, 2020, 12:27 AM UTC

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.