Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kafka Removing Zookeeper Dependency

confluent.io

Post reply on HN
Print view View original

161–170 of 183 posts

Re: Kafka Removing Zookeeper Dependency

#161

Post by manigandham »

StreamBright wrote: post #151
agallego wrote: post #146

Earlier quoted context omitted.

I understand tail latencies - https://www.youtube.com/watch?v=WdFYY3vEcxo - my prev open source project smf ( https://github.com/smfrpc/smf ) uses gill tene's HDR histogram. I think you have a very superficial understanding of seastar. It is not simply marketing, it is a suite of tools and techniques to build low latency software - Specifically for IO intensive apps. Glauber wrote a good into here https://www.scyllad…

I am familiar with Seastar too. It is one component that is pretty useless by itself. What is relevant in this topic is what is around it, the functionality that you provide. This is why Scylla is copying Cassandra. You can come up with a nice way of programming whatever you want but the end of the way the business functionality is what matters and there are different tradeoffs involved, still.

What do you mean "copying" Cassandra? Obviously they're offering the same API. Many people like the Cassandra data model and multiregional capabilities and that's why it was chosen.

What Scylla is doing is unlocking new performance potential with a C++ rewrite and an entirely different process-per-core architecture that gets around the fundamental limitations of Cassandra and makes it easier to run. This performance and stability has also led to the team making existing C* features like LWT, secondary indexes, and read-repair even faster and better than the original implementations.

Re: Kafka Removing Zookeeper Dependency

#162

Post by Groxx »

pjmlp wrote: post #160
Groxx wrote: post #158

Earlier quoted context omitted.

I honestly have yet to see any evidence that Fuchsia is anything but an experiment in this respect, like Dart was (which was also going to "replace android"). I entirely admit the possibility , but I see no plan . Hopes and aspirations are not plans.

> That being said, if you were writing Android from scratch and were only targeting ARM (an equally silly comparison meant to highlight the differences in the languages), you’d be hard pressed to justify Java over Go. Instead they went with Rust, C++ and Dart. Now what have all those languages in common that Go lacks?

I think you meant to reply to the parent comment with this?

Re: Kafka Removing Zookeeper Dependency

#163

Post by maxdo »

agallego wrote: post #21
tbrock wrote: post #11

Finally! Travis Jeffery did this years ago in Jocko and also solved my other beef with Kafka at the same time by building it in Golang. https://github.com/travisjeffery/jocko I’ve always found things built on JVM are are PITA to deploy (especially when using SSL) so the single Golang binary is a welcome advancement. It’s all the good things about Kafka (concept, API, and wire protocol) without all the crap (zookeeper…

Agreed. We're working on this exactly - https://vectorized.io/redpanda/ 10x faster. API compat. No jvm. I also know of other private impls of it. Just makes sense.

No sources, no benchmarks, no production use-cases. Thanks but no.

Re: Kafka Removing Zookeeper Dependency

#164

Post by maxdo »

StreamBright wrote: post #145
manigandham wrote: post #138

Earlier quoted context omitted.

Easier for who? At some point, the architecture needs to change to overcome a fundamental limit. Developers take on that work to make the performance and operations easier for their users. Seastar is the foundation of Scylla, which shows that rewriting in C++ can deliver magnitudes more performance which is not possible by just tuning Cassandra on the JVM. In fact, Datastax has now copied the Scylla approach in Cassa…

>> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in…

How does Scylla relates to Kafka and zookeeper here? I know a bunch of ad-tech companies that built their entire stack around JVM and scala. Those companies need to perform a bid (a whole bunch of logic ) within 100 ms ( including networking), otherwise, they got a financial penalty. Please stop being a kid who post everywhere you see JVM product your opinion on that. Kafka, Pulsar are two successful projects no matter what you think. Kafka relies on a bit outdated architecture but has the most vibrant ecosystem. Those are major pros and cons. Not a language or VM.

Re: Kafka Removing Zookeeper Dependency

#165

Post by throwdbaaway »

manigandham wrote: post #159
StreamBright wrote: post #145

Earlier quoted context omitted.

>> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in…

Yes, you get 10x the throughput while maintaining far better tail latency with Scylla over Cassandra. How is 10ms vs 475 not a major improvement? How is 4 nodes vs 40 not a major improvement? If you're an SRE than how is managing 4 servers with far less tuning and maintenance not a major improvement? Also 99.9% percentile still matters. They're testing with 300k ops/sec which means 300/sec are facing extreme latency…

2-3x maybe, 10x not likely, unless you compare it to untuned cassandra.

What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java.

So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 big boxes, don't forget about TWCS.

Re: Kafka Removing Zookeeper Dependency

#166

Post by manigandham »

throwdbaaway wrote: post #165
manigandham wrote: post #159

Earlier quoted context omitted.

Yes, you get 10x the throughput while maintaining far better tail latency with Scylla over Cassandra. How is 10ms vs 475 not a major improvement? How is 4 nodes vs 40 not a major improvement? If you're an SRE than how is managing 4 servers with far less tuning and maintenance not a major improvement? Also 99.9% percentile still matters. They're testing with 300k ops/sec which means 300/sec are facing extreme latency…

2-3x maybe, 10x not likely, unless you compare it to untuned cassandra. What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java. So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 b…

It's 4 servers doing the same as 40. That is 10x throughput, and with lower tail latency.

Scylla is far more than a compaction strategy. If it was that simple, than Cassandra would already be able to do it.

It's an objectively faster database in every metric. Datastax's enterprise distribution has more functionality but core Cassandra is now entirely outclassed by Scylla in speed and features.

Re: Kafka Removing Zookeeper Dependency

#167

Post by manigandham »

maxdo wrote: post #164
StreamBright wrote: post #145

Earlier quoted context omitted.

>> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in…

How does Scylla relates to Kafka and zookeeper here? I know a bunch of ad-tech companies that built their entire stack around JVM and scala. Those companies need to perform a bid (a whole bunch of logic ) within 100 ms ( including networking), otherwise, they got a financial penalty. Please stop being a kid who post everywhere you see JVM product your opinion on that. Kafka, Pulsar are two successful projects no matt…

Did you reply to the wrong post?

Redpanda uses the Seastar framework which was created by the ScyllaDB project. Scylla is high-performance C++ reimplemation of Cassandra and RedPanda seems to be chasing the same thing as an alternative to Kafka/JVM.

As a 12 year adtech veteran who has built ad networks from scratch 3 times, low-latency and high-throughput are critical to ad serving infrastructure and that's why Scylla is such a better alternative to Cassandra. The only other database that gets close is Aerospike, and possibly Redis Enterprise with Flash persistence. It's entirely valid to want similar improvements for event streams as well, and as long as they keep the same external API then you don't lose any of the ecosystem advantages either.

Re: Kafka Removing Zookeeper Dependency

#168

Post by vips7L »

bsaul wrote: post #152
throwdbaaway wrote: post #142

Earlier quoted context omitted.

You do realize that golang statically compiled binaries contain the go runtime right? So, all the concerns that you mentioned are applicable to go as well, e.g. a golang app may be buggy when built with version X, while another golang app may have GC issue when built with version Y, and yet another golang app would be vulnerable unless rebuilt with version Z. [1][2][3] Don't drink the Kool-Aid, golang is just a more…

except that the developer controls it, since its is embedded. As such it follows the exact same lifecycle as the code itself ( versionning, testing, etc).

This is misinformed. jlink has existed since jdk 11 and allows the developer to produce a custom JVM to ship with their application. jpackage also now exists to create installers for whatever platform your on. It is you the developer who are responsible for packaging your application correctly.

Re: Kafka Removing Zookeeper Dependency

#169

Post by throwdbaaway »

manigandham wrote: post #166
throwdbaaway wrote: post #165

Earlier quoted context omitted.

2-3x maybe, 10x not likely, unless you compare it to untuned cassandra. What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java. So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 b…

It's 4 servers doing the same as 40. That is 10x throughput, and with lower tail latency. Scylla is far more than a compaction strategy. If it was that simple, than Cassandra would already be able to do it. It's an objectively faster database in every metric. Datastax's enterprise distribution has more functionality but core Cassandra is now entirely outclassed by Scylla in speed and features.

Again, that's 40 mid size boxes with questionable GC (32G to 48G heap size is the no man land, G1GC target pause time of 500ms of course will result in 500ms P999 latency, etc), versus 4 big boxes, which have more than 4x higher specs. So 10 divides by 4, that's the 2-3x that I mentioned.

The TWCS is just a good example that raw performance is not everything. Performance also comes from things like compaction strategy, data modeling, and access pattern, while users and stakeholders also care about things like easiness to modify, friendly license, and steady stewardship.

Re: Kafka Removing Zookeeper Dependency

#170

Post by throwdbaaway »

agallego wrote: post #147
StreamBright wrote: post #145

Earlier quoted context omitted.

>> Easier for who? For any decent SRE out there. >> magnitudes more performance which is not possible by just tuning Cassandra Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first. https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben... Avg. 99.9% Latency (ms): 9.9 vs. Avg. 99.9% Latency (ms): 474.4 While there is no significant latency difference in…

who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. https://www.youtube.com/watch?v=lJ8ydIuPFeU What matters is what most of your customers will get - p99, p999, p9999, p100.

Ironically, Gil Tene uses the talk to sell you the Azul C4 pauseless GC, which shall easily invalidate your 10x lower p99 latency claim. Of course we also have zgc and shenandoah these days.
Post reply on HN