Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

121–130 of 201 posts

Re: Kafka without ZooKeeper

#121

Post by tsimionescu »

shaklee3 wrote: post #118
cmckn wrote: post #16

Earlier quoted context omitted.

Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group). NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;) N…

> NATS doesn't ever store messages persistently Not true. Both Nats streaming and the upcoming jetstream (core nats) do.

It depends what you mean by 'persistently'. Normally NATS streaming will delete the messages after they have been delivered to all subscribers successsfully and some expiration time has passed.

Re: Kafka without ZooKeeper

#122

Post by airhead969 »

rrdharan wrote: post #98
godisdad wrote: post #51

Earlier quoted context omitted.

Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind

The irony of the name Cloudera has not been lost on keen observers of this space.

It makes you wonder how many people think "clouds" run on something besides servers.

Re: Kafka without ZooKeeper

#123

Post by EdwardDiego »

rad_gruchalski wrote: post #111
EdwardDiego wrote: post #108

Earlier quoted context omitted.

Yep, I have, no issues with the consumer/producer knowing who the partition leader is. That said, curious to hear your experiences :)

I don’t have issues per se as long as I stick to librdkafka but even that is constantly playing catch up. Outside of librdkafka and jvm client, it’s gloves off.

Yeah, that's true, using librdkafka from C# I hit a few issues where librdkafka was somewhat behind Java in terms of features, I think the one I hit was multi-topic subscriptions.

IIRC Confluent has started putting resources into it - I would hope so, given how .NET Core is going.

That said, the state of Pulsar clients outside of the official Java ones was far worse, I was looking into .NET Core ones and the "official" one (Pulsar-DotPulsar) lacked some key features, whereas a third party one, pulsar-client-dotnet, had far more features, but was still somewhat behind the Java clients.

Caveat is that I looked into all of this when Pulsar was at version 2.6, it's not at 2.7.1, so my comments may well be out of date.

Re: Kafka without ZooKeeper

#124

Post by EdwardDiego »

rad_gruchalski wrote: post #112
EdwardDiego wrote: post #106

Earlier quoted context omitted.

It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO. And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.

> offload old data into S3 and then retrieve it if required I remember asking for it 5 years ago: https://radek-gruchalski.medium.com/the-case-for-kafka-cold-... . Confluent turned it into a paid feature.

Yep, it's available from Confluent if you pay for it, but like how Mirror Maker 2 is awfully similar to Confluent Replicator, I believe that Kafka will (eventually) get tiered storage under the Apache licence (I know there's a KIP for it[1]). It's a hard issue to solve, and not sure how much effort in the community is being directed towards it. But bear in mind that it's not just Confluent who have a stake in Kafka - there's a bunch of big corps selling managed/supported Kafka and all of them would probably quite like tiered storage in core Kafka as a feature to help them sell their support/management, so I have some faith in their enlightened self-interest.

The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith.

Incidentally, I quite like how Pulsar solved tiered storage, and it's a definite tick in the Pulsar box - it's transparent from a consumer's POV, although there somewhat of a delay in rehydrating the offloaded block, I don't think anyone's expecting near-realtime performance when loading historical data.

[1]: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...

Re: Kafka without ZooKeeper

#125

Post by mrweasel »

haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

For people who just need a queue, Kafka is a bit like using Kubernetes to run a single Docker container.

We run a number of Kafka clusters, most are relatively low trafic, and the management overhead is pretty. Earlier version did require a bit more attention, but mostly it’s pretty simple to deal with.

Re: Kafka without ZooKeeper

#126

Post by mavelikara »

nemothekid wrote: post #65
doliveira wrote: post #40

Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.

Before Raft (2013), if you wanted reliable, consistent distributed metadata store you had to implement Paxos which is notoriously difficult to get right. Every service that needed some type of leader election or highly consistent store let Zookeeper deal with that problem (Mesos, Spark, Druid, Storm, and a ton others). After Raft, it became easier to just implement that layer yourself and so most projects after Raft…

What are the notable projects that implement Raft internally for leader election?

Also, do any of those projects publish their Raft implementation as a library for other projects to include?

Re: Kafka without ZooKeeper

#127

Post by beberlei »

mavelikara wrote: post #126
nemothekid wrote: post #65

Earlier quoted context omitted.

Before Raft (2013), if you wanted reliable, consistent distributed metadata store you had to implement Paxos which is notoriously difficult to get right. Every service that needed some type of leader election or highly consistent store let Zookeeper deal with that problem (Mesos, Spark, Druid, Storm, and a ton others). After Raft, it became easier to just implement that layer yourself and so most projects after Raft…

What are the notable projects that implement Raft internally for leader election? Also, do any of those projects publish their Raft implementation as a library for other projects to include?

Anything from hashicorp, vault, consul, nomad for example. yes there is a go library for the basic raft setup afaik

Re: Kafka without ZooKeeper

#129

Post by manigandham »

taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

There's actually no reason to choose Pulsar anymore. Pulsar has even more layers with Zookeeper + Bookkeeper that requires something like Kubernetes to run well. It was great 5 years ago for heavy users who need better scalability and features than Kafka, however the development has become a mess.

With the removal of zookeeper and tiered storage (separated from compute), Kafka has caught up on scalability while being simpler to deploy. It also has a far bigger ecosystem with more polished features like ksqldb.

Post reply on HN