Live data from Hacker News

Kafka without ZooKeeper

confluent.io

11–20 of 201 posts

Re: Kafka without ZooKeeper

#11
post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

[deleted]

Re: Kafka without ZooKeeper

#12
post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.

Re: Kafka without ZooKeeper

#15

One immediate downside to this: it is now likely that fewer and fewer people will be exposed to the absolutely excellent Zookeeper logo.

Is there some backstory on this? It looks like it's modelled after a real person. Perhaps one of its creators?

Re: Kafka without ZooKeeper

#16
post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

NATS isn't actually using a log structure though, it's a streaming message broker with a different set of consistency/delivery promises.

Correct; but I've seen many uses of Kafka that NATS could totally be used for. For example, load balancing across subscribers (use a NATS queue instead of a Kafka consumer group).

NATS doesn't ever store messages persistently; but this might be fine for your application, and then you don't have to worry about setting 5 different config options to make sure Kafka actually frees up disk space like you expect it to ;)

NATS also enables some unique patterns like request/reply via a "reply to" message header.

Anyway, it's been a joy to use!

Re: Kafka without ZooKeeper

#17
post #6
post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/

https://vectorized.io/redpanda/ is a more useful link, since the main domain appears to have some JS errors right now.

Re: Kafka without ZooKeeper

#18
post #2

Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D

You can tune Kafka down fairly well if you know what you're doing, but it's not optimised for that OOTB. Or just use Confluent Cloud, which is fully managed and scales down as low as you want (costs cents per Gb). Disclosure: work for Confluent.
Post reply on HN