Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

161–170 of 201 posts

Re: Kafka without ZooKeeper

#161

Post by sumtechguy »

colin_mccabe wrote: post #57
haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to man…

Oh it most certainly simplifies things. I am looking at half the number of boxes needed to run. Which is not insignificant in my cost structure.

What is the migration strategy here? Is it doc'd up yet? I am having flashbacks to migration for follower partitions recently which required a decent amount of pre planning of partition layout.

Also as it is pulling in the duties of ZK into kafka what sort of CPU/memory changes are you seeing? Is it 'meh' or all the way to 'you may want to add a couple of CPUs and a few more GB'? Also is it working ok with the stretched cluster?

Also if you want to hit an interesting market you may want to look at 'does it run OK on a raspberry PI'.

Re: Kafka without ZooKeeper

#162

Post by agallego »

thibauts wrote: post #145
didip wrote: post #73

Reminded me of this project: https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.

I’m trying to take a similar path with this project [1] though there’s no HA story for now and it’s not meant to be protocol-compatible. The goal is to make it lightweight and simple to operate, yet very fast. [1] https://github.com/dataptive/styx

From a feature list perspective on the readme it basically is what we did at Redpanda (dev here)

Re: Kafka without ZooKeeper

#163

Post by nvarsj »

nemothekid wrote: post #65
doliveira wrote: post #40

Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.

Before Raft (2013), if you wanted reliable, consistent distributed metadata store you had to implement Paxos which is notoriously difficult to get right. Every service that needed some type of leader election or highly consistent store let Zookeeper deal with that problem (Mesos, Spark, Druid, Storm, and a ton others). After Raft, it became easier to just implement that layer yourself and so most projects after Raft…

Raft is a large improvement over Paxos for practical implementations. But it's still tricky to get right. As far as I know, the only widely used, battle tested Raft implementation is github.com/hashicorp/raft. Which is why so many distributed systems are being built on golang over the last few years. I don't know if there is any Java raft implementation which has reached that level of maturity yet - but it seems like Confluence is trying with kraft.

Re: Kafka without ZooKeeper

#164

Post by shaklee3 »

tsimionescu wrote: post #121
shaklee3 wrote: post #118

Earlier quoted context omitted.

> NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.

It depends what you mean by 'persistently'. Normally NATS streaming will delete the messages after they have been delivered to all subscribers successsfully and some expiration time has passed.

The expiration time can be unlimited, just like kafka

Re: Kafka without ZooKeeper

#165

Post by tuhaihe »

atombender wrote: post #92
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Apache Pulsar is hardly an ideal comparison in this context, considering that Pulsar requires ZooKeeper and Apache BookKeeper (which also requires ZooKeeper). One of the benefits of the Kafka rearchitecture effort is to allow Kafka to "scale down" to run without external dependencies. Using Pulsar would add more dependencies.

As I know, Pulsar community is doing the work of removing ZooKeeper, this is a helpful link: https://github.com/apache/pulsar/projects/10

Re: Kafka without ZooKeeper

#166

Post by SEJeff »

ric2b wrote: post #160
the_arun wrote: post #113

Earlier quoted context omitted.

Agreed. We need highly available & distributed 1. Locking service 2. Id generator 3. Bloom filter etc.,

Why the id generator? Aren't GUID's enough? The chance of a collision is basically 0.

A few different points of view:

https://tomharrisonjr.com/uuid-or-guid-as-primary-keys-be-ca...

https://news.ycombinator.com/item?id=14523523 (the hacker news comments for the above article are really insightful)

https://www.percona.com/blog/2019/11/22/uuids-are-popular-bu...

https://blog.codinghorror.com/primary-keys-ids-versus-guids/

Re: Kafka without ZooKeeper

#167

Post by atombender »

tuhaihe wrote: post #165
atombender wrote: post #92

Earlier quoted context omitted.

Apache Pulsar is hardly an ideal comparison in this context, considering that Pulsar requires ZooKeeper and Apache BookKeeper (which also requires ZooKeeper). One of the benefits of the Kafka rearchitecture effort is to allow Kafka to "scale down" to run without external dependencies. Using Pulsar would add more dependencies.

As I know, Pulsar community is doing the work of removing ZooKeeper, this is a helpful link: https://github.com/apache/pulsar/projects/10

They are, but I think they have some ways to go still. BookKeeper is still on ZooKeeper, though they've abstracted the API and have added support for using Etcd instead (not sure if this is production-ready).

Re: Kafka without ZooKeeper

#168

Post by rad_gruchalski »

EdwardDiego wrote: post #159
rad_gruchalski wrote: post #147

Earlier quoted context omitted.

> Okay, I can see that point, but is it worth the additional latency between broker and Bookie? It might depend on what you're ingesting and how much. Being able to independently scale ingest and storage is a good alternative to have. It's not only ingest though. It's also consumption. As it stands, having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must…

> having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory. I don't know too much about Kafka's internals, but that's my not experience of reading several terabytes of data from a Kafka topic. Memory didn't blow out, although we did burn through IOPs credits.

Good remark. When there's one consumer or multiple consumers hanging on roughly to the same offset (using the same memory mapped segment). Having many consumers hanging on widely different offsets will cause many segments sitting in RAM.

Edit: Kafka apparently does not store a complete log segment in memory, only parts but having many consumers may lead to a lot of churn or a lot of memory consumed. Maybe this is getting better.

Re: Kafka without ZooKeeper

#169

Post by mdaniel »

sgt wrote: post #131
gigatexal wrote: post #128

All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/ )

What's the downside?

The BSL license makes choosing it "a discussion" versus using OSI licensed software: https://github.com/vectorizedio/redpanda/tree/v21.3.7/licens...

I don't have enough experience with these new vanity licenses to know what the contribution story looks like, either

Re: Kafka without ZooKeeper

#170

Post by the_arun »

ric2b wrote: post #160
the_arun wrote: post #113

Earlier quoted context omitted.

Agreed. We need highly available & distributed 1. Locking service 2. Id generator 3. Bloom filter etc.,

Why the id generator? Aren't GUID's enough? The chance of a collision is basically 0.

For most of the use cases GUID works great. There are some cases where we need a numerical id which is sequential.
Post reply on HN