All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/ )
What's the downside?
Kafka without ZooKeeper
131–140 of 201 posts
Re: Kafka without ZooKeeper
#132Earlier quoted context omitted.
It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.
Agreed. We need highly available & distributed 1. Locking service 2. Id generator 3. Bloom filter etc.,
Azures blob service supports the first one. Its strongly consistent and has a lease/lock API. Ive seen leader election and coordination stuff built on top of it.
Re: Kafka without ZooKeeper
#133I don't understand the hate for Zookeeper. Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.
Sure, a basic ensemble isn't that hard. But securing it is a world of pain and misery.
The ZK development API is also pretty much awful. Apache Curator (which wraps around the ZK API and implements a bunch of common "recipes") makes it less painful, but it really ought to be part of ZK proper.
Re: Kafka without ZooKeeper
#134Re: Kafka without ZooKeeper
#135I don't understand the hate for Zookeeper. Setting up a Zookeeper ensemble is not that hard, they're light on resources and and basically zero maintenance.
Sure, a basic ensemble isn't that hard. But securing it is a world of pain and misery. The ZK development API is also pretty much awful. Apache Curator (which wraps around the ZK API and implements a bunch of common "recipes") makes it less painful, but it really ought to be part of ZK proper.
God forbid you hit a bug in ZK too, as there are 5 people in the world who properly understand that system - there is nobody to help.
Re: Kafka without ZooKeeper
#136Earlier quoted context omitted.
Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent
I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.
> You end up splitting your data across partitions anyways
Messages are ordered within partitions.
> What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.
TCP/IP ?
Re: Kafka without ZooKeeper
#137Earlier quoted context omitted.
1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?
Re 2, have you ever tried using Kafka with a non-JVM client?
kafkacat is excellent.
Re: Kafka without ZooKeeper
#138Is there a compiled package for 2.8.0-rc0 available? I only see sources on Github https://github.com/apache/kafka/releases/tag/2.8.0-rc0 and couldn't find anything regarding 2.8.0-rc0 on the Kafka downloads page https://kafka.apache.org/downloads .
The downloads page is only for official releases, RCs are not distributed the same way and are not permanent unless promoted to an official release.
See the mailing list thread for 2.8.0-RC0 for where to find the bits if you want to test https://lists.apache.org/thread.html/r16894a11aec73abac521ff... and the project site has some "contact" info for mailing lists where these things are announced and advertised (including releases, Kafka Improvement Proposals, and more) https://kafka.apache.org/contact
Re: Kafka without ZooKeeper
#139Reminded me of this project: https://github.com/travisjeffery/jocko Kafka implemented in Go without needing Zookeeper.
Hey, I'm the author of Jocko. I've been working at Confluent the past four years.
I just finished writing a book that shows how to build similar distributed services from scratch, it walks though building a simple distributed commit log with built-in consensus and service discovery from nothing to deployment: https://pragprog.com/titles/tjgo/distributed-services-with-g...
Re: Kafka without ZooKeeper
#140Great news! Props!
I have a use case where this would make sense, I’ll dig in directly!