This is cool. One thing I love about Cassandra is that you don't need Zookeeper, its just a bunch of homogeneous nodes.
Same with Elastic Search.
Kafka Removing Zookeeper Dependency
31–40 of 183 posts
Re: Kafka Removing Zookeeper Dependency
#32Earlier quoted context omitted.
Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to...
They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.
yep so not 100% the same, but my point with linking to it is tunables / levers / control is useful. In other scenarios maybe you could get by without having to do a rewrite in Rust, maybe you could keep it in golang and tune the GC a bit.
Whilst I like the memory ballast as a clever solution, it does smell like a workaround to a solvable problem if a parameter existed to tune it directly.
Re: Kafka Removing Zookeeper Dependency
#33Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…
Someone is adding journaling to the NATS protocol to make it Kakfa like:
It was just released as v1.0 a few weeks ago.
Re: Kafka Removing Zookeeper Dependency
#34Earlier quoted context omitted.
They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.
yep so not 100% the same, but my point with linking to it is tunables / levers / control is useful. In other scenarios maybe you could get by without having to do a rewrite in Rust, maybe you could keep it in golang and tune the GC a bit. Whilst I like the memory ballast as a clever solution, it does smell like a workaround to a solvable problem if a parameter existed to tune it directly.
It's totally a workaround and I'm jumping on that last comment when I shouldn't be; it's just a reaction to the hyperbolic "you might have to rewrite" conclusion, because you won't have to rewrite, but indeed you might be annoyed by the lack of tunables.
Re: Kafka Removing Zookeeper Dependency
#35I never know where I sit on stuff like this. On the one hand if you’re confluent I think it makes total sense to own this part of your infrastructure. Especially if it lets you improve your operability story. On the other hand I feel like projects should try and use open source “building-blocks”, like etcd and zookeeper, when building their distributed systems. Not only does this help iron out correctness bugs, but i…
If ZK wasn't such a pain in the ass to manage I'd agree with you.
Re: Kafka Removing Zookeeper Dependency
#36Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…
Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.
Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.
Re: Kafka Removing Zookeeper Dependency
#37Earlier quoted context omitted.
Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.
Nice job. I have always wondered why performant big data systems are written using resource hungry JVM.
Thanks! Yeh I think the same, specially with the rise of core count. I think an i3.metal is at 96vcpus. I estimate a machine like that should hold around 400K partitions just fine with us. I should measure this weekend.
Re: Kafka Removing Zookeeper Dependency
#38Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…
Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.
@agallego I just signed up on your wait list through the vectorized.io website. I'm super interested in this project.
Re: Kafka Removing Zookeeper Dependency
#39Earlier quoted context omitted.
Sometimes having more control over GC is useful tho, could save a re-write. https://blog.discord.com/why-discord-is-switching-from-go-to...
They rewrote to get away from garbage collection full-stop, so I don't think this is quite the argument for the JVM that you're making it out to be. Not to mention: they resolved their Go GC problem, and rewrote the service in Rust as part of a wave of moving services to Rust because they just liked Rust.
I'm an active Go user and I've been working almost exclusively in Go since 2011. I have a feeling that over time, the number of knobs on Go will increase much like it has in Java. I'm trying to think back to my first experiences with Java in 1995. It was so simple compared to the overhead of getting things working in C/C++. So Java's got like 25+ years on Go and I would speculate that the number of knobs on Go in 25 years will be significantly higher. On the one hand, I don't like seeing all the crazy configurable stuff, e.g. CGO_....=1 until I actually need it.
Re: Kafka Removing Zookeeper Dependency
#40Earlier quoted context omitted.
Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.
@agallego I just signed up on your wait list through the vectorized.io website. I'm super interested in this project.
on it.