Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

51–60 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#51

Post by skyde »

whorleater wrote: post #35
hugofirth wrote: post #4

I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…

If ZK wasn't such a pain in the ass to manage I'd agree with you.

this pain come from paxos raft or Zab algorithm used by concensus system. I don’t think it can be avoided.

Re: Kafka Removing Zookeeper Dependency

#52

Post by agallego »

skyde wrote: post #46
agallego wrote: post #21

Earlier quoted context omitted.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

so this use raft for topic data not just metadata ?

correct. we use one protocol, just raft - both for metadata and for data.

Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained -https://twitter.com/emaxerrno/status/1260415381321084929

Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and not w/ the protocol. so it gives users sound reasoning.

Re: Kafka Removing Zookeeper Dependency

#53

Post by skyde »

lchengify wrote: post #42

I couldn't be more excited about this! Having finished a Kafka project about a year ago, we had so many Zookeeper production and test environment issues that it was the running joke to check Zookeeper first if anything went wrong. Honestly Zookeeper in theory is a great idea: Having a centralized service for maintaining config info saves a lot of heartache when dealing with an open source distributed systems project.…

i am curious to know why people expect a raft library to be more reliable if embedded inside the kafka controller versus running inside a service like Etcd.

in the end broker will do RPC to a service (kafka controller/ etcd) and this service will use raft to replicate the state.

It should be exactly the same. And if anything knowing which node are running the raft algorithm help you be more careful with rolling restart and upgrade.

Re: Kafka Removing Zookeeper Dependency

#54

Post by skyde »

agallego wrote: post #52
skyde wrote: post #46

Earlier quoted context omitted.

so this use raft for topic data not just metadata ?

correct. we use one protocol, just raft - both for metadata and for data. Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained - https://twitter.com/emaxerrno/status/1260415381321084929 Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and no…

are you hiring :-) I would love to work on this.

I consider myself a RAFT expert and worked in kafka for the past 5 years.

Re: Kafka Removing Zookeeper Dependency

#55

Post by pjmlp »

Twirrim wrote: post #44
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's a black box optimizer for databases, which is designed to take the user away from having to find and tweak all the hundreds of possible tunables in the database, https://github.com/cmu-db/ottertune / https://ottertune.cs.cmu.edu/ I keep wanting the same for JVM. It's probably nice that there are so many ways to adjust the JVM to best suit a very wide range of workloads, but from a practical perspective I rare…

Well, ART auto tunes and probably not everyone will agree it does it perfectly.

There are also plenty JVMs to pick and some are better than others working with their defaults.

Re: Kafka Removing Zookeeper Dependency

#56

Post by DiabloD3 »

tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

All you're doing is exchanging one runtime, the JVM, that happens to be well understood, highly optimized, and has a decade long track record, and supported by every major company on Earth... for Go's runtime, one that isn't well understood, does not have a track record, and is a product of a company that is known to just abandon popular services for no apparent reason.

You also seem to have misunderstood what causes some apps to be a pain to deploy: the apps, themselves, suck, and would suck no matter what language they had been written in. Apache tends to become the home for a lot of projects that exhibit that particular issue, and Zookeeper does not seem to be an exception.

Kafka likely also has a case of the Apacheisms going on, but also apparently has a non-trivial amount of Scala in it. Complaining about Kafka fits your requirements far more than Zookeeper alone does.

Re: Kafka Removing Zookeeper Dependency

#57

Post by agallego »

skyde wrote: post #54
agallego wrote: post #52

Earlier quoted context omitted.

correct. we use one protocol, just raft - both for metadata and for data. Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained - https://twitter.com/emaxerrno/status/1260415381321084929 Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and no…

are you hiring :-) I would love to work on this. I consider myself a RAFT expert and worked in kafka for the past 5 years.

alex@vectorized.io - let's chat. i'm looking for a senior member atm.

Re: Kafka Removing Zookeeper Dependency

#58

Post by bobbyi_settv »

vips7L wrote: post #30
tbrock wrote: post #16

Earlier quoted context omitted.

Setting a min and max heap size for a garbage collected language in 2020. Just figure it out for me?

There's nothing to figure out. If you want to use all your ram (like in go) just set -Xmx .

But to know whether you want that, you need to understand the application. Elasticsearch, for example, recommends using at most 50% of physical memory for heap:

https://www.elastic.co/guide/en/elasticsearch/reference/curr...

You also often want to stay below 32 gb to get compressed OOPs

Re: Kafka Removing Zookeeper Dependency

#59

Post by silisili »

elric wrote: post #48
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

Re: Kafka Removing Zookeeper Dependency

#60

Post by sz4kerto »

snidane wrote: post #36
agallego wrote: post #21

Earlier quoted context omitted.

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.

Because the JVM is fast. And it's not resource hungry - your program might be resource hungry. Bad desktop Java programs misled a whole generation of programmers about the JVM.

Look at stuff like LMAX. Java can be lightning fast.

Post reply on HN