Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

121–130 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#121

Post by jjirsa »

welder wrote: post #114
jjirsa wrote: post #14

Earlier quoted context omitted.

Why throw away something proven to run at massive scale, that you understand and trust for something that's new, has never been run at that scale, and you have no experience running? If you have a team of software engineers, and the latency problem is a software problem, fix the software problem. When you already know Cassandra, and you already know RocksDB, and you already have an engineering team, it makes far more…

> some new thing NOBODY has run at scale Outbrain uses ScyllaDB in production at scale across multiple data centers. Not sure if it's Instagram scale, but still enough to prove it's reliability and performance. https://www.outbrain.com/techblog/2016/08/scylladb-poc-not-s...

7 hosts in that poc, that is not "at scale"

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#122

Post by jjirsa »

threeseed wrote: post #92
coryfoo wrote: post #33

Earlier quoted context omitted.

I bet that didn't take N engineers 12 months to build out, either

Cassandra uses G1GC by default. If it was as simple as tweaking a few GC settings to get 10x improvement pretty sure Datastax would've done it by now.

The problem with making a general purpose DB is you have to have general purpose defaults. The Cassandra defaults are "dont crash anywhere", not "be super fast and low latency". You can definitely do 5x better than the default with some basic jvm tuning.

That said: the IG folks certainly know how to tune JVMs. There are IG (and former IG, I saw rbranson post) folks in this thread that know how to tune the collectors, so assume that the 10x they see is beyond what you'd get from simple tuning.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#125

Post by nitsanw »

rbranson wrote: post #23
en4bz wrote: post #19

Has any tried running Casandra on Azul Zing[1]? The slowdown here is not surprisingly related to GC pauses which Azul has eliminated in Zing. [1] https://www.azul.com/products/zing/

The licensing cost of Zing generally makes this a bad trade-off. It's much cheaper to purchase more hardware. Zing is targeted at vertically scaling very large JVM heaps, where it's valuable to have massive amounts of data on a single, big machine.

As an ex-Azul employee I can say there's a good number of Azul clients using a Zing+Cassandra setup, so the price point is right for some people at the very least. Zing licence cost has also changed in recent years (3.5k per server last I looked, and that is before you haggle some bulk deal) so not sure if your impression is calibrated to that new price point.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#127

Post by chii »

thefounder wrote: post #99
nvarsj wrote: post #98

Earlier quoted context omitted.

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Apparently these people enjoy GC/JVM languages more than C++.

GC languages like Java is much easier to write, and can be made performant when required.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#128

Post by katastic »

b4lancesh33t wrote: post #101
_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latenc…

Let me guess. I can feel it coming. You're about to segway into talking about Rust?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#129

Post by majidazimi »

mschuster91 wrote: post #44
en4bz wrote: post #41

Earlier quoted context omitted.

That removes the value proposition of Java though which is that you don't need to worry about memory management. If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++.

> If you need to mentally track every implicit allocation and deallocation in Java then you are essentially writing code in a kneecapped version of C++. Well, Java has the advantage of being platform (and to a certain degree, runtime) independent, plus a robust set of best practices and ecosystem when it comes to modules and library handling, which is pretty hard to get done right for C/C++ projects.

> Well, Java has the advantage of being platform (and to a certain degree, runtime) independent

What is the benefit of that? Who on earth runs a DB written in Java on windows? Any useful server software will end up using platform native features, be it SQL server, MySQL, HBase, ...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#130

Post by majidazimi »

cestith wrote: post #69
adrianratnapala wrote: post #60

As a Java scoffer trying to be fair-minded, I resisted the urge to joke that "it's was the GC, stupid" and assume that a big project like Cassandra had somehow worked around the GC latency problems. But, what? It turns out the article is really about replacing Java with C++.

It's about using something in one language for its features and only porting the critical sections to C++ via a clean API. This is the sort of advice we've been giving people for decades. Choose the language for what you want to build, measure and profile performance if necessary, find the bottleneck on the hot path, decouple that from the bulk of the code, and reach to a lower level for performance only in that clea…

Well, other than storage engine, the next big part of a database software is the query planner/optimizer which Cassandra doesn't have (due to simple KV nature of it). So there isn't much remaining. In a long term plan, rewrite them all and you have single code base and you'll benefit from mighty C++ in other components of the database. And there is still room for more optimizations: SIMD, ...

The GC problem is not limited to C*. This shit(virtual machine) is hitting the whole Hadoop stack: HDFS, Hive, Spark, Flink, Pig...

Immense number of tickets in any fairly large cluster is related somewhat to GC and JVM behavior.

Post reply on HN