Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka without ZooKeeper

confluent.io

Post reply on HN
Print view View original

91–100 of 201 posts

Re: Kafka without ZooKeeper

#91

Post by taywrobel »

EdwardDiego wrote: post #87
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

As someone who evaluated Pulsar to replace Kafka, my thoughts... More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication. Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the…

StreamNative doesn’t lock critical functionality behind enterprise agreements while still advertising the software as open source, and doesn’t openly lie about what the system capabilities are.

Not saying that they won’t turn evil at some point, but so far they’re leagues ahead of Confluent in terms of earning developer trust. At a minimum this developer, but also others that I’ve worked with.

Maybe I’m the minority opinion here and that’s fine, but confluent has been far too shady for me to ever consider contracting with them.

Re: Kafka without ZooKeeper

#92

Post by atombender »

taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

Apache Pulsar is hardly an ideal comparison in this context, considering that Pulsar requires ZooKeeper and Apache BookKeeper (which also requires ZooKeeper).

One of the benefits of the Kafka rearchitecture effort is to allow Kafka to "scale down" to run without external dependencies. Using Pulsar would add more dependencies.

Re: Kafka without ZooKeeper

#93

Post by nuharaf »

gantengx wrote: post #62
cmckn wrote: post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent

I believe with jetstream, message in stream is ordered as they are written. Jetstream have a concept of consumer, (in the broker itself, not client), which can consume a subset of the stream, filtered by message subject.

Re: Kafka without ZooKeeper

#94

Post by tammerk »

nemothekid wrote: post #71
doliveira wrote: post #63

Earlier quoted context omitted.

I guess what I wonder is why they didn't go with an embedded library or something of sorts. Some NoSQL databases handle it without Zookeeper.

> Some NoSQL databases handle it without Zookeeper. Most NoSQL databases, now, use Raft, which didn't exist at the time when Kafka was created. Other NoSQL databases, at the time, were not as stable as Zookeeper or had silent bugs that ate data (see aphyr's Jepsen series[1], which thourghly tested several NoSQL databases and found many to be failing, except for Zookeeper). [1] https://aphyr.com/tags/jepsen

https://github.com/jepsen-io/jepsen/issues/399

> Yeah! I mean, I find a lot of linearizability errors in various databases, but this was also my very first time doing this kind of test, and it varies from system to system. Could have easily slipped through the cracks.

In summary, aphyr thought Zookeeper is linearizable even though it doesn't provide linearizable ops.

Looks like Zookeeper needs to be tested again.

Re: Kafka without ZooKeeper

#95

Post by closeparen »

airhead969 wrote: post #3

IIRC, ZK would be more modern and cloud-friendly if it could self-assemble with a preshared passphrase alone. It's good technology otherwise, it's just a PITA to deploy, configure, and support.

It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.

Re: Kafka without ZooKeeper

#96

Post by agallego »

timdorr wrote: post #17
dmlittle wrote: post #6

Earlier quoted context omitted.

Haven't used it personally myself but I've heard it enough to remember it. Redpanda[1] aims to be a Kafka replacement without having to worry about Zookeeper or the JVM [1] https://vectorized.io/

https://vectorized.io/redpanda/ is a more useful link, since the main domain appears to have some JS errors right now.

oh odd. what setup to repro the js errors. i'll fix.

Re: Kafka without ZooKeeper

#97

Post by staticassertion »

gantengx wrote: post #62
cmckn wrote: post #8

Oh hell yeah! That's great news, tons of work went into this -- props to the contributors! I will take the opportunity to say that Kafka is kind of painful, with or without ZK. Check out NATS! [0]. It doesn't solve all the same problems, but is so much easier to use (during development especially) and can do a lot of the same things. [0]: https://docs.nats.io/whats_new_20

Is there any message ordering guarantee in NATS? With Kafka you can achieve this by using keyed message and messages in the same partition will always be ordered Disclosure: I work for Confluent

I've never really understood the appeal of ordered messages. You end up splitting your data across partitions anyways for parallelism, so who cares? What systems out there require strictly ordered data? It seems like any design that requires something like that is going to be extremely brittle.

Re: Kafka without ZooKeeper

#98

Post by rrdharan »

godisdad wrote: post #51
cmckn wrote: post #9

Earlier quoted context omitted.

I really love a lot of the software under the Hadoop umbrella, but so much of it assumes a static deployment on bare metal hosts, it's a struggle to use it in "modern" setups (HBase, for example; I miss my old friend).

Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind

The irony of the name Cloudera has not been lost on keen observers of this space.

Re: Kafka without ZooKeeper

#99

Post by rad_gruchalski »

ovis wrote: post #59
skyde wrote: post #43

Earlier quoted context omitted.

I think Pulsar is a much better design and further investment in Kafka is a mistake at this point. Kafka have a lot of downside 1- size for single topic limited to the size of one machine 2- complex stateful client library that need to know which machine is currently the master for each partition. ....

1. That's untrue. Partitions are limited to what a machine may handle, but topics may be scaled across many ordered partitions. 2. This is generally handled by the client library transparently. Have you ever needed to manage this state manually?

Re 2, have you ever tried using Kafka with a non-JVM client?

Re: Kafka without ZooKeeper

#100

Post by [deleted] »

EdwardDiego wrote: post #87
taywrobel wrote: post #41

If designing a new system is there any reason to choose Kafka over Pulsar at this point? Apart from Confluent wanting you to use Kafka so they can keep leeching money off you by hijacking de facto ownership of an open source project, of course.

As someone who evaluated Pulsar to replace Kafka, my thoughts... More moving parts. Brokers, and Bookies, and ZK, plus proxies etc. Plus an additional ZK for inter-cluster replication. Immaturity - it's still early days for Pulsar, and there's still a lot of bugs being found - and then rapidly fixed, full credit to them, but yeah, not yet as stable. Documentation is often obsoleted, and I found myself having read the…

[deleted]
Post reply on HN