Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

101–110 of 201 posts

Re: Kafka without ZooKeeper

#101

Post by rad_gruchalski »

tveita wrote: post #80
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. (That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

> Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka.

BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.

Re: Kafka without ZooKeeper

#102

Post by z9e »

dikei wrote: post #82

I don't understand the hate for Zookeeper. Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.

Agreed. I’ve been running Kafka clusters at a few companies now, one that was at a massive Fortune 500 and Zookeeper was the least of my problems.

I’m still glad to see it go away, one less operational dependency the better.

Re: Kafka without ZooKeeper

#103

Post by EdwardDiego »

taywrobel wrote: post #91
EdwardDiego wrote: post #87

Earlier quoted context omitted.

As someone who evaluated Pulsar to replace Kafka, my thoughts... More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication. Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the…

StreamNative doesn’t lock critical functionality behind enterprise agreements while still advertising the software as open source, and doesn’t openly lie about what the system capabilities are. Not saying that they won’t turn evil at some point, but so far they’re leagues ahead of Confluent in terms of earning developer trust. At a minimum this developer, but also others that I’ve worked with. Maybe I’m the minority…

What critical functionality are you thinking of?

We used FOSS Kafka for yonks without hitting any limitations - At one point we were looking at Confluent Replicator, but decided it was just easier to go with Mirror Maker 1 (and you know, no massive licensing fees) - and Mirror Maker 2 largely emulates Replicator in terms of functionality.

I'm aware of a few other things like the MQTT KC Connector, but that was never part of Kafka in the first instance, it's something that Confluent built for paying customers.

And I could argue that the StreamNative "critical functionality" that they lock away behind enterprise agreements is "quick bug fixes for the many bugs we're still finding", if I was feeling mean spirited.

But anyway, it seems your preference for Pulsar is due to Confluent, but they're not the only ones offering managed Kafkas - AWS, IBM, RedHat, etc. etc.

Re: Kafka without ZooKeeper

#104

Post by EdwardDiego »

knodi wrote: post #58

Please separate storage from brokers next

Why though?

It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka.

https://blog.twitter.com/engineering/en_us/topics/insights/2...

> One catch to this is that for extremely bandwidth-heavy workloads (very high fanout-reads), EventBus theoretically might be more efficient since we can scale out the serving layer independently. However, we’ve found in practice that our fanout is not extreme enough to merit separating the serving layer, especially given the bandwidth available on modern hardware.

Re: Kafka without ZooKeeper

#105

Post by manigandham »

cmckn wrote: post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

NATS is just ephemeral at-most-once pub/sub. It needs NATS Streaming or the new Jetstream for at-least-once persisted data and still has different semantics.

Apache Pulsar offers the same distributed log offering with a fundamentally better architecture, but Kafka has closed most of the gaps now and has far more integrations and a bigger ecosystem.

Re: Kafka without ZooKeeper

#106

Post by EdwardDiego »

rad_gruchalski wrote: post #101
tveita wrote: post #80

Earlier quoted context omitted.

Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. (That said, unlike many I consider depending on ZooKeeper to be a positive sign. "We wrote our own consensus protocol" belongs in roughly the same bucket as "we wrote our own crypto." Using ZooKeeper doesn't automatically mean your distributed system will work but at least you'll have a fighting chance.)

> Pulsar also requires you to run ZooKeeper and BookKeeper, so TFA has at least one reason you might choose Kafka. BookKeeper is a feature though. Allows to scale the partition beyond the capacity of a storage unit. Effectively unlimited retention for a partition. The problem with Kafka is that the broker is tied to storage.

It can be a feature, yep, although Pulsar's tiered storage (offload old data into S3 and then retrieve it if required) is far more of a feature IMO.

And BK isn't a feature without cost, it's documentation is... somewhat sparse, and there is significant complexity to maintaining it.

Re: Kafka without ZooKeeper

#107

Post by EdwardDiego »

taywrobel wrote: post #48
tyingq wrote: post #44

Earlier quoted context omitted.

Hijacking seems like a strong word since Jay Kreps is the CEO.

Fair enough, not hijacking. Just presenting the software as open source then requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker). Add to that their insistence in claiming “exactly once delivery semantics” from Kafka despite that being provably impossible and I don’t see any reason to trust them as a company or pay…

> requiring you to pay substantially for features that it’d be irresponsible to use the software in production without (e.g. geo replication via mirror maker).

MM1 and MM2 are free. You might be getting confused with Confluent Replicator.

> Kafka compatible adapters

For very limited subsets of the Kafka APIs.

Re: Kafka without ZooKeeper

#108

Post by EdwardDiego »

rad_gruchalski wrote: post #99
ovis wrote: post #59

Earlier quoted context omitted.

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

Re 2, have you ever tried using Kafka with a non-JVM client?

Yep, I have, no issues with the consumer/producer knowing who the partition leader is.

That said, curious to hear your experiences :)

Re: Kafka without ZooKeeper

#109

Post by nick0garvey »

EdwardDiego wrote: post #104
knodi wrote: post #58

Please separate storage from brokers next

Why though? It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka. https://blog.twitter.com/engineering/en_us/topics/insights/2... > One catch to this is that for extremely bandwidth-heavy…

Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot.

I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

Re: Kafka without ZooKeeper

#110

Post by EdwardDiego »

nick0garvey wrote: post #109
EdwardDiego wrote: post #104

Earlier quoted context omitted.

Why though? It's worth noting that Twitter built their own system (EventBus) that Apache Pulsar largely mimics in design (and the people who started Pulsar at Yahoo had worked on EventBus prior), with brokers decoupled from storage, and then eventually just decided to get rid of it and use Kafka. https://blog.twitter.com/engineering/en_us/topics/insights/2... > One catch to this is that for extremely bandwidth-heavy…

Some workloads are very CPU intensive and some are not. Being forced to scale CPU & disk together means one of them is going to be overprovisioned - often by a lot. I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

Okay, I can see that point, but is it worth the additional latency between broker and Bookie?

> I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases.

Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.

Post reply on HN