Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

71–80 of 201 posts

Re: Kafka without ZooKeeper

#71

Post by nemothekid »

doliveira wrote: post #63
anonymousDan wrote: post #52

Earlier quoted context omitted.

Because historically implementing something like Zookeeper yourself from scratch is notoriously difficult?

I guess what I wonder is why they didn't go with an embedded library or something of sorts. Some NoSQL databases handle it without Zookeeper.

>Some NoSQL databases handle it without Zookeeper.

Most NoSQL databases, now, use Raft, which didn't exist at the time when Kafka was created. Other NoSQL databases, at the time, were not as stable as Zookeeper or had silent bugs that ate data (see aphyr's Jepsen series[1], which thourghly tested several NoSQL databases and found many to be failing, except for Zookeeper).

[1] https://aphyr.com/tags/jepsen

Re: Kafka without ZooKeeper

#72

Post by cmckn »

andrewmutz wrote: post #56
cmckn wrote: post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

Does NATS support (or plan to support) log compaction? Without that, it's very hard to replace Kafka.

I"m not sure I understand your question; NATS streaming server (built on top of NATS) supports persistence to disk, a raft group, a SQL table, etc. and it appears the various storage implementations have mechanisms [0] to delete or compress old data.

That being said, I don't think this is what differentiates the two systems, the guarantees they do/don't make are likely what will make the decision for your project.

[0]: https://github.com/nats-io/nats-streaming-server/blob/master...

Re: Kafka without ZooKeeper

#74

Post by d_t_w »

alex_anglin wrote: post #60
math wrote: post #18

Earlier quoted context omitted.

You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.

Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?

Confluent Cloud has some nice point-and-click UI for creating associated Kafka resources like Schema Registries and Connect Clusters.

My preference is MSK but I'm very comfortable with vanilla Kafka in AWS at a good price with auto-updates.

Re: Kafka without ZooKeeper

#75

Post by Hallucinaut »

colin_mccabe wrote: post #69
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

I personally think Kafka has the edge in many ways. It will soon be possible to run a single-process Kafka cluster, which will unlock a lot of applications that previously people used an older systems for, simply because it was easier than standing up a full ZK cluster + Kafka cluster. The broader Kafka ecosystem has features like exactly-once support, KSQL, Kafka Connect, Cluster Linking, and excellent client suppor…

What challenges do you see Pulsar having (and potentially not overcoming)?

Re: Kafka without ZooKeeper

#77

Post by dikei »

nemothekid wrote: post #65
doliveira wrote: post #40

Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.

Before Raft (2013), if you wanted reliable, consistent distributed metadata store you had to implement Paxos which is notoriously difficult to get right. Every service that needed some type of leader election or highly consistent store let Zookeeper deal with that problem (Mesos, Spark, Druid, Storm, and a ton others). After Raft, it became easier to just implement that layer yourself and so most projects after Raft…

To be fair, many project's Raft implementations contained errors that can and had lead to data lost, so it's not all sunshine and roses.

IMHO, it's still easier to delegate the consensus problem to a third party service like Zookeeper or ETCD.

Re: Kafka without ZooKeeper

#78

Post by pdimitar »

colin_mccabe wrote: post #57
haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to man…

Your clarification made me wonder:

Is the single process deployment only doable via a container? Or will we actually have OS native process as well?

Re: Kafka without ZooKeeper

#79

Post by tamale »

cmckn wrote: post #16
vorpalhex wrote: post #12

Earlier quoted context omitted.

NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.

Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) N…

you also don't have to worry about those kinds of configuration gotchas if you use confluent cloud!

Re: Kafka without ZooKeeper

#80

Post by tveita »

taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka.

(That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

Post reply on HN