Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

171–180 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#171

Post by elric »

skywhopper wrote: post #127
elric wrote: post #48

Earlier quoted context omitted.

Things on the JVM are a PITA to deploy? That's a bit of a silly statement. Some things are easy to deploy, some aren't. It's not an intrinsic property of the JVM, but rather the consequence of a series of choices made by whoever wrote the tool in question. You can do TLS in Java without using keystores, even if keystores do have some advantages, and megacorps seem to like them for all the wrong reasons. Using them sh…

Wow, this does not match my experience at all. The JVM itself is harder to install than any native-binary service I've ever used. And that's before you get around to running something like Kafka or Zookeeper on top of it. Maybe "not all things" are a pain to deploy, but everything I've ever run on it or looked into running has been. I'm particularly curious about how you deal with TLS in the JVM without keystores? I'…

How are you installing these troublesome JVMs? Either you use your OS' package manager and get updates as a bonus (for a looooong time on CentOS/RHEL). Or you can literally download an OpenJDK zipfile, unzip it and you're done -- will require manual updating. Or, if you're feeling enterpricey (not a typo), you can install an Oracle JVM and get paid updates for an eternity.

Kafka and ZK are a pain to deploy. Not the JVM's fault. Entirely down to the choices made by their respective community. Last I checked it was still virtually impossible to secure a ZK-ensemble. Maybe they'd welcome patches, but when I tried to submit a minor bugfix years ago, I found them to be an unwelcoming community.

Re: Kafka Removing Zookeeper Dependency

#172

Post by [deleted] »

throwdbaaway wrote: post #169
manigandham wrote: post #166

Earlier quoted context omitted.

It's 4 servers doing the same as 40. That is 10x throughput, and with lower tail latency. Scylla is far more than a compaction strategy. If it was that simple, than Cassandra would already be able to do it. It's an objectively faster database in every metric. Datastax's enterprise distribution has more functionality but core Cassandra is now entirely outclassed by Scylla in speed and features.

Again, that's 40 mid size boxes with questionable GC (32G to 48G heap size is the no man land, G1GC target pause time of 500ms of course will result in 500ms P999 latency, etc), versus 4 big boxes, which have more than 4x higher specs. So 10 divides by 4, that's the 2-3x that I mentioned. The TWCS is just a good example that raw performance is not everything. Performance also comes from things like compaction strateg…

[deleted]

Re: Kafka Removing Zookeeper Dependency

#173

Post by pjmlp »

Groxx wrote: post #162
pjmlp wrote: post #160

Earlier quoted context omitted.

> That being said, if you were writing Android from scratch and were only targeting ARM (an equally silly comparison meant to highlight the differences in the languages), you’d be hard pressed to justify Java over Go. Instead they went with Rust, C++ and Dart. Now what have all those languages in common that Go lacks?

I think you meant to reply to the parent comment with this?

Yeah I thought it was the same person.

Re: Kafka Removing Zookeeper Dependency

#174

Post by pjmlp »

manigandham wrote: post #161
StreamBright wrote: post #151

Earlier quoted context omitted.

I am familiar with Seastar too. It is one component that is pretty useless by itself. What is relevant in this topic is what is around it, the functionality that you provide. This is why Scylla is copying Cassandra. You can come up with a nice way of programming whatever you want but the end of the way the business functionality is what matters and there are different tradeoffs involved, still.

What do you mean "copying" Cassandra? Obviously they're offering the same API. Many people like the Cassandra data model and multiregional capabilities and that's why it was chosen. What Scylla is doing is unlocking new performance potential with a C++ rewrite and an entirely different process-per-core architecture that gets around the fundamental limitations of Cassandra and makes it easier to run. This performance…

While exposing a memory corruption friendly language into the internet.

What kinds of security assessments are done to guarantee that Scylla is as secure as Cassandra?

Re: Kafka Removing Zookeeper Dependency

#175

Post by parasubvert »

elric wrote: post #88
silisili wrote: post #59

Earlier quoted context omitted.

I don't understand Java apologists. I'm not saying Java is bad. Quite the opposite, it's one of the most optimized systems to date. But if you cannot admit Java apps are a pain to deploy, or, at least harder than deploying a static binary, I cannot take this argument seriously in good faith.

Seriously? When was the last time you deployed something as a standalone, static binary? Hardly anything with any amount of complexity is shipped like that, especially not cross-platform. Just about everything requires some kind of runtime, some kind of dependencies or some kind of environment. I can go "dnf install firefox" just as easily as I can go "dnf install ant". Both of these come with a boatload of dependenc…

Nearly everything written in Golang is deployed as a static binary with minimal dependencies, which is partly why it has grown in popularity.

It’s a nearly universal experience that different language runtimes have very different experiences with dependencies. Someone who ships a Node, Python or Ruby-based tool for example has a tough time, to the point of needing to write a wrapper installer that vendors all dependencies including the runtime itself, just to be sure.

The JVM with Maven and fat JARs isn’t probably as bad as these cases, as mostly it’s just requiring a compatible JRE. This is why Spring Boot apps for example are so popular for deployment - no more app server insanity!

Jlink makes this more like Golang, but isn’t used enough.

That said Golang dependency management for the developer / builder is a history of horrors. Maven has had its issues but has gotten past most of them.

Re: Kafka Removing Zookeeper Dependency

#176

Post by parasubvert »

ashtonkem wrote: post #141
apta wrote: post #132

Earlier quoted context omitted.

You can ship the JVM with your app such that it's self-contained and system independent. Apps like IntelliJ already do this. And now with jlink, you can strip it down to make deployment sizes even smaller by only shipping the parts of the JVM that your app uses.

That’s the reasonable choice for consumer facing applications, yeah. For server applications it makes more sense to use something like Docker or a pre-built AMI. For what I do “how do I get Java on the server” is much less difficult than making deployments quicker and more efficient, which is more a problem for our CI/CD harness and integration tests. Neither of which are Java specific per se.

For server apps you don’t even need a Docker image or AMI, a fat JAR will usually do 90% of what you need.

That said if you want to keep your JDK up to date via Docker or AMI of course that’s fine. a jlink JAR does the same thing but I can see the desire for wanting to decouple runtime upgrades from code upgrades.

Re: Kafka Removing Zookeeper Dependency

#177

Post by skyde »

lima wrote: post #112
skyde wrote: post #49

Earlier quoted context omitted.

100 % agree we need good (etch/zookeeper) as a library. I wrote something similar but that provide only (lock/lease) using paxos. The problem with raft ETCD and zookeeper replicates state machine design is that : - machine leaving and joining the ensemble dynamically is something hard to do correctly - optimal quorum size is no more than 5, you can setup other node as observer but it’s hard to decide which of 1000 no…

The etcd server can be embedded in other projects, and their Raft library is used by plenty of other projects like CockroachDB.

yes it can but it’s a real pain compared to hashicorp version. I really hope they clean up the library to have a simple API.

This way it could become the standard for GO and we can stop wasting effort on multiple raft implementation.

Re: Kafka Removing Zookeeper Dependency

#178

Post by manigandham »

pjmlp wrote: post #174
manigandham wrote: post #161

Earlier quoted context omitted.

What do you mean "copying" Cassandra? Obviously they're offering the same API. Many people like the Cassandra data model and multiregional capabilities and that's why it was chosen. What Scylla is doing is unlocking new performance potential with a C++ rewrite and an entirely different process-per-core architecture that gets around the fundamental limitations of Cassandra and makes it easier to run. This performance…

While exposing a memory corruption friendly language into the internet. What kinds of security assessments are done to guarantee that Scylla is as secure as Cassandra?

Sure, that's a valid but separate concern. You'd have to ask the dev team what their precise security approach is.

Re: Kafka Removing Zookeeper Dependency

#179

Post by tamale »

femiagbabiaka wrote: post #149
Tsarbomb wrote: post #114

Earlier quoted context omitted.

Good luck when you need to deploy to a system with a different version of libc.

goalposts are moving at a rapid pace

having a different version of libc is exactly like worrying that the system has a different version of java. If you know about these 'gotchas', they're not big deal. If you don't, then ya, they can be a bit intimidating at first.

I really don't think running a go app vs. a java app is ANY different.

Re: Kafka Removing Zookeeper Dependency

#180

Post by femiagbabiaka »

tamale wrote: post #179
femiagbabiaka wrote: post #149

Earlier quoted context omitted.

goalposts are moving at a rapid pace

having a different version of libc is exactly like worrying that the system has a different version of java. If you know about these 'gotchas', they're not big deal. If you don't, then ya, they can be a bit intimidating at first. I really don't think running a go app vs. a java app is ANY different.

I’m trying to figure out what environments you’re working in where libc incompatibilities are as common as java versioning issues

You’ve made a straw man and are now working very hard to defend it.

Post reply on HN