Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

51–60 of 201 posts

Re: Kafka without ZooKeeper

#51

Post by godisdad »

cmckn wrote: post #9
airhead969 wrote: post #3

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).

Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind

Re: Kafka without ZooKeeper

#52

Post by anonymousDan »

doliveira wrote: post #40

Finally. I assume there must be good reasons beyond "that's what Hadoop has always used" but philosophically, I never understood why introduce yet another network dependency to handle elections. It really adds up to the operational complexity, from having to manage the Zookeeper cluster to having to fight against DNS.

Because historically implementing something like Zookeeper yourself from scratch is notoriously difficult?

Re: Kafka without ZooKeeper

#53

Post by dan_quixote »

taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Quoting myself here [1], but Kafka is still very architecturally un-sound for containerized deployments.

-------------------------------------------------------

The problem I see with kafka is that it was built before cloud architectures were commonly adopted (with distributed systems everywhere). Confluent has put a lot of effort dragging kafka's architecture to the present, but some major features are missing:

  - Auto-scaling:  Confluent finally introduced "elastic scaling" a few months ago but it only allows you to scale up and must be triggered by the admin (no threshold-based auto-scaling).

  - Multi-tenancy:  Planning for a multi-tenant kafka cluster is not for the faint of heart.  Achieving isolation tends toward liberal usage of topics of which starts to become unmanageable in the low thousands.  This isn't crazy when you've got a few hundred microservices and several tenants to keep isolated.

  - Decoupled brokers and storage:  Any broker scaling or failure can lead to downtime while event storage is redistributed.
Confluent's Cloud service reduces operational overhead but isn't always feasible due to cost, resource limits (like service accounts or schemas for instance), data controls, etc.

-------------------------------------------------------

[1] https://news.ycombinator.com/item?id=25984560

Re: Kafka without ZooKeeper

#54

Post by plank_time »

taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

I’ve been following Kafka for 5+ years now. Confluent has done their level best to stack the Kafka PMC with their own employees. Only recently have they added other members. They should not be an Apache project due to how much gaming of ownership they are doing with the project.

Re: Kafka without ZooKeeper

#56

Post by andrewmutz »

cmckn wrote: post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

Does NATS support (or plan to support) log compaction? Without that, it's very hard to replace Kafka.

Re: Kafka without ZooKeeper

#57

Post by colin_mccabe »

haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness."

Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode).

For bigger projects, you may want to use a managed cloud service. Or if you do choose to manage it yourself, it will be easier running one service than two.

Disclosure: I work for Confluent.

Re: Kafka without ZooKeeper

#59

Post by ovis »

skyde wrote: post #43
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions.

2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

Re: Kafka without ZooKeeper

#60

Post by alex_anglin »

math wrote: post #18
haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.

Why would someone choose Confluent Cloud over the Kafka offerings of Azure/AWS/GCP?
Post reply on HN