Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

21–30 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#21

Post by agallego »

tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Agreed. We're working on this exactly - https://vectorized.io/redpanda/

10x faster. API compat. No jvm.

I also know of other private impls of it. Just makes sense.

Re: Kafka Removing Zookeeper Dependency

#22

Post by staticassertion »

tbrock wrote: post #16
closeparen wrote: post #14

Earlier quoted context omitted.

Keystores are clunky, but other than that, why exactly?

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

But Go doesn't "figure it out" for you. It decides for you - there's no "figure it out" step.

That's the power of having tunables.

Re: Kafka Removing Zookeeper Dependency

#23

Post by nhoughto »

tbrock wrote: post #16
closeparen wrote: post #14

Earlier quoted context omitted.

Keystores are clunky, but other than that, why exactly?

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

Sometimes having more control over GC is useful tho, could save a re-write.

https://blog.discord.com/why-discord-is-switching-from-go-to...

Re: Kafka Removing Zookeeper Dependency

#24

Post by manigandham »

agallego wrote: post #21
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?

Re: Kafka Removing Zookeeper Dependency

#25

Post by tptacek »

nhoughto wrote: post #23
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to...

They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be.

Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.

Re: Kafka Removing Zookeeper Dependency

#26

Post by agallego »

manigandham wrote: post #24
agallego wrote: post #21

Earlier quoted context omitted.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

Nice, always like seeing new projects. Is this based on Seastar? Will this be open-source?

Yeah. Based on seastar.

W.r.t OSS doing research on it atm.

I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision

Re: Kafka Removing Zookeeper Dependency

#27

Post by chucky_z »

saurik wrote: post #12
sagichmal wrote: post #8

Earlier quoted context omitted.

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

I feel like this just argues that ZooKeeper should be a library that can be embedded in the servers of other projects (even if it is conceptually a separate server in the same memory space listening on its own ports; but like, it is otherwise entirely hidden inside of the other program and is entirely managed and configured by it also).

Isn't this exactly what lead to Hashicorp's memberlist library being used all over in random projects?

Re: Kafka Removing Zookeeper Dependency

#28

Post by halbritt »

sagichmal wrote: post #8
hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

Counterpoint, if I need to deploy technology T to solve problem P, I don't want to have to also deploy a flotilla of support technologies because modularity or whatever. ZooKeeper was always an implementation detail of Kafka, the fact that you had to manage it separately was an abstraction leak that I'm happy to see fixed.

The fact that many people are scared away from Kafka simply because they don't want to run zookeeper should be telling.

Re: Kafka Removing Zookeeper Dependency

#29

Post by biggestlou »

nealabq wrote: post #10

I expect Pulsar to stay with Zookeeper. Kafka currently stores topic and partition info on ZK ( https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data... ), which can get to be a lot of data. But I think Pulsar only stores server names and basic config info on ZK ( https://pulsar.apache.org/docs/en/administration-zk-bk/ ), which is much more managable.

That is correct: Pulsar uses ZooKeeper relatively lightly and only for things strictly necessary for the cluster to run. Everything else (message data, topic info, cursors, schemas) is stored in BookKeeper.

Would still be great to see ZooKeeper made superfluous, though.

Re: Kafka Removing Zookeeper Dependency

#30

Post by vips7L »

tbrock wrote: post #16
closeparen wrote: post #14

Earlier quoted context omitted.

Keystores are clunky, but other than that, why exactly?

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx.
Post reply on HN