Kafka without ZooKeeper

#1

Post by toomanybits » Tue, Mar 30, 2021, 8:35 PM UTC

Re: Kafka without ZooKeeper

#2

Post by haolez » Tue, Mar 30, 2021, 9:14 PM UTC

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Re: Kafka without ZooKeeper

#3

Post by airhead969 » Tue, Mar 30, 2021, 9:21 PM UTC

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

Re: Kafka without ZooKeeper

#4

Post by autophagian » Tue, Mar 30, 2021, 9:29 PM UTC

One immediate downside to this: it is now likely that fewer and fewer people will be exposed to the absolutely excellent Zookeeper logo.

Re: Kafka without ZooKeeper

#5

Post by cmason » Tue, Mar 30, 2021, 9:29 PM UTC

haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

What do you use instead?

Re: Kafka without ZooKeeper

#6

Post by dmlittle » Tue, Mar 30, 2021, 9:30 PM UTC

haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM

[1] https://vectorized.io/

Re: Kafka without ZooKeeper

#7

Post by spicybright » Tue, Mar 30, 2021, 9:30 PM UTC

autophagian wrote: post #4

One immediate downside to this: it is now likely that fewer and fewer people will be exposed to the absolutely excellent Zookeeper logo.

I did a google image search and wish I didn't.

Re: Kafka without ZooKeeper

#8

Post by cmckn » Tue, Mar 30, 2021, 9:35 PM UTC

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors!

I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things.

[0]: https://docs.nats.io/whats_new_20

Re: Kafka without ZooKeeper

#9

Post by cmckn » Tue, Mar 30, 2021, 9:38 PM UTC

airhead969 wrote: post #3

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).

Re: Kafka without ZooKeeper

#10

Post by [deleted] » Tue, Mar 30, 2021, 9:44 PM UTC

dmlittle wrote: post #6

haolez wrote: post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/

[deleted]