Don't forget that ZooKeeper was the only service that passed all Jepsen [1] tests out of the box.
Kafka without ZooKeeper
151–160 of 201 posts
Re: Kafka without ZooKeeper
#152Earlier quoted context omitted.
Cheap managed cloud services, like AWS SQS and Azure Storage Queue (I usually want some kind of persistence for my queues).
Confluent Cloud Basic/Standard is a cheap managed Kafka. If the objection is to the deployment and not Kafka clients.
We might have different definitions of cheap.
Re: Kafka without ZooKeeper
#153Kafka is a pretty cool technology, but for every project that I work on, it's never used because it feels like it's overkill (costly and operation heavy). Maybe I should start looking for bigger projects :D
Part of the reason we are removing Kafka's ZooKeeper dependency is to get rid of that "heaviness." Going forward, you will no longer need to configure and run a separate ZooKeeper service just to run Kafka. For proof-of-concept projects, a single-process Docker image will be available when running in KRaft mode (non-ZK mode). For bigger projects, you may want to use a managed cloud service. Or if you do choose to man…
A nit regarding the disclosure: I prefer it at the top of the message, and I think that's "best practice", but I don't know for sure.
Re: Kafka without ZooKeeper
#154All this talk about NATs as an alternative to Kafka with no mention of Redpanda’s Zookeeper-less alternative (written in modern C++ - https://vectorized.io/ )
What's the downside?
Doesn’t seem to be one. They took the Kafka API so it’s a drop in replacement and redid the implementation from the ground up. Your mileage may vary though.
Re: Kafka without ZooKeeper
#155Earlier quoted context omitted.
Yeah. They should have written it twenty years ago in a datacenter with public cloud in mind
Hadoop launched in 2006, the same year as AWS' cloud portfolio. HBase showed up in 2008. Many of the hiccups with running Hadoop and friends in containers or on cloud VM's boils down to how hostnames are resolved and advertised; not any significant design issue.
> Hadoop launched in 2006, the same year as AWS' cloud portfolio.
Which makes it all the less reasonable to assume it would be in any way cloud native, when "the cloud" was at best a nascent idea at that point.
And how many years did it take AWS to get any serious traction after launch?
Re: Kafka without ZooKeeper
#156Earlier quoted context omitted.
What's the downside?
Doesn’t seem to be one. They took the Kafka API so it’s a drop in replacement and redid the implementation from the ground up. Your mileage may vary though.
I am just worries that things like 3rd parties integrating with your Kafka broker will stop working or experience issues. Also, I wonder if SASL_SSL is implemented exactly the same way etc.
Re: Kafka without ZooKeeper
#157Cool feature!
I have a concern, if the Kafka broker provide both coordination service and Kafka service, how to achieve the resource isolation? If some of the topic which on the coordination service with very high throughput, this must cause instability of the coordination service, could this further cause the instability of the entire cluster?
If some of the broker only provide the coordination service, what is this essential difference? Will this cause more problems for expansion and contraction? Will this bring greater risks when users scale down the brokers? I am very afraid that the coordination service will be shut down due to careless operation.
Re: Kafka without ZooKeeper
#158Earlier quoted context omitted.
Yep, it's available from Confluent if you pay for it, but like how Mirror Maker 2 is awfully similar to Confluent Replicator, I believe that Kafka will (eventually) get tiered storage under the Apache licence (I know there's a KIP for it[1]). It's a hard issue to solve, and not sure how much effort in the community is being directed towards it. But bear in mind that it's not just Confluent who have a stake in Kafka -…
Thanks for putting things in perspective, EdwardDiego. > The fact that MM2 happened, and Confluent didn't try to stop it, despite it being awfully similar to Replicator, makes me think that Confluent are acting in good faith. Let me share an anecdote related to this example. We (Confluent) were actually the ones who contributed the documentation for MirrorMaker v2 to the Apache Kafka docs ( https://kafka.apache.org/d…
Shit, wait, there's documentaton for Mirror Maker 2 now? I spent most of my time implementing it by reading hypothetical examples in a KIP, and then diving into the actual code.
Hardly the most straightforward, and it was rather a gaping hole. Thanks for the background on how that hole developed.
I really appreciate Confluent putting that time into documenting something vital, that could compete with your own product, and IMO that does put a nail in the previous commenter's assertions about Confluent's alleged attempts to wall off necessary features of Kafka.
Re: Kafka without ZooKeeper
#159Earlier quoted context omitted.
Okay, I can see that point, but is it worth the additional latency between broker and Bookie? > I'm pretty surprised Twitter didn't see benefit from doing this if they have multiple Kafka clusters with different use cases. Yeah, I think they were too tbh. I wish I could delve more into what they experienced beyond that single blog post I linked.
> Okay, I can see that point, but is it worth the additional latency between broker and Bookie? It might depend on what you're ingesting and how much. Being able to independently scale ingest and storage is a good alternative to have. It's not only ingest though. It's also consumption. As it stands, having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must…
> having a parallel consumer over a large partition spanning several GBs also requires tons of RAM because a segment must be loaded into memory.
I don't know too much about Kafka's internals, but that's my not experience of reading several terabytes of data from a Kafka topic. Memory didn't blow out, although we did burn through IOPs credits.
Re: Kafka without ZooKeeper
#160Earlier quoted context omitted.
It seems weird to me that cloud providers do not offer distributed coordination primitives "as a service." I understand there are KV stores but not with watches, locks, etc. in the way that etcd and ZK have them.
Agreed. We need highly available & distributed 1. Locking service 2. Id generator 3. Bloom filter etc.,
Why the id generator? Aren't GUID's enough? The chance of a collision is basically 0.