Open-sourcing a 10x reduction in Apache Cassandra tail latency • Hacker News Board

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#151

Post by MaxBarraclough » Tue, Mar 06, 2018, 5:21 PM UTC

tibbetts wrote: post #118

itronitron wrote: post #102

Earlier quoted context omitted.

developing in Java with awareness of the GC doesn't mean tracking allocation and deallocation of memory, it means developers should avoid allocating lots of new Objects when possible. In practice this means creating view, cursor, or offset type Objects that map to arrays of more primitive data types.

Exactly. Developing GC aware code is still easier and safer than c++ memory management. Particularly because when you screw up in c++ you get crashes or data loss, and when you screw up in Java you mostly just get GC pauses.

Mandatory mention of D and its @nogc feature (as it sounds, a compile-time guarantee that a function doesn't allocate on the garbage-collected heap)

https://dlang.org/blog/2017/06/16/life-in-the-fast-lane/

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#152

Post by OhDagny » Tue, Mar 06, 2018, 5:21 PM UTC

dikanggu wrote: post #34

We do want to contribute our work back to the Cassandra upstream, instead of keeping it as a fork. So that more users from C* community can benefit from the improvements. The pluggable storage engine is an ambitious project ( https://issues.apache.org/jira/browse/CASSANDRA-13474 ). Any help will be appreciated!

Fundamentally, the "pluggable storage" is probably the driver protocol. As in Scylla, Cassandra, or Cass/Rocks

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#153

Post by amarkov » Tue, Mar 06, 2018, 5:40 PM UTC

majidazimi wrote: post #131

amarkov wrote: post #46

Earlier quoted context omitted.

Even in a database, most of the code isn't performance sensitive. Making your life a bit harder in the fast path so it's easier in the slow path is at least a tradeoff worth considering.

Based on previous research [1], almost every part of a DB is in the hot path. [1] http://15721.courses.cs.cmu.edu/spring2018/papers/02-inmemor...

I'd definitely believe that every part of query execution is in the hot path, as this paper describes. But most code in any reasonable DB system isn't part of query execution.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#154

Post by sreque » Tue, Mar 06, 2018, 6:08 PM UTC

b4lancesh33t wrote: post #101

_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latenc…

You're right on your first point. I responded a little harshly to OP because I believe they responded too harshly to the blog post. The P99 GC latency data they give is us not sufficient to explain their perf gains. More likely, the metric they were tracking was well-correlated with their perf gains, but not the only or even primary cause. The other perf gains could have been caused by reduction in older generation collection times, for instance.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#155

Post by domsj » Tue, Mar 06, 2018, 7:12 PM UTC

manigandham wrote: post #139

truth_seeker wrote: post #117

Earlier quoted context omitted.

Curious to know the cons of using it, except being commercial.

> except being commercial That's the biggest, especially for when it's a Facebook company. Otherwise it works well but can be pricey. The JVM is getting a new fully concurrent collector though called Shenandoah: https://www.google.com/search?q=shenandoah+gc

Not just shenandoah[1], it's also getting zgc[2], so the low latency future looks bright for the jvm.

1. https://wiki.openjdk.java.net/display/shenandoah/Main 2. https://wiki.openjdk.java.net/display/zgc/Main

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#156

Post by the8472 » Tue, Mar 06, 2018, 7:58 PM UTC

eptcyka wrote: post #143

b4lancesh33t wrote: post #101

Earlier quoted context omitted.

> Given that stop-the-world pauses are never evenly distributed throughout time That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latenc…

I'd love to see how Go's GC performs when running an application similar to Cassandra, on multiple cores, with gigabytes of memory allocated.

Go's GC is non-moving, incremental and single-generation. Hotspot GCs are all moving collectors so they make different tradeoffs. IBM's metronome collector would be the closest one to Go's properties.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#157

Post by the8472 » Tue, Mar 06, 2018, 8:01 PM UTC

nvarsj wrote: post #98

_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

OpenJDK/Hotspot is not the only game in town. Those who really need low latencies can opt to use other JVMs (some commercial) with GCs that provide very low pause times, usually at the expense of some percentage points of throughput. In large corporate environments that might not be a problem.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#158

Post by teacpde » Tue, Mar 06, 2018, 8:36 PM UTC

the8472 wrote: post #86

teacpde wrote: post #80

Earlier quoted context omitted.

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

No, that would be an incremental GC working in very small time slices. A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. They are still CPU cycles you don't have to serve other requests, hence they still affect throughput. That is a simplified explanation of course, there are a lot of caveats. In my original post…

> A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests.

That makes sense, how about GC for single-threaded languages, e.g. Nodejs?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#159

Post by geofft » Tue, Mar 06, 2018, 9:38 PM UTC

en4bz wrote: post #59

geofft wrote: post #48

Earlier quoted context omitted.

Instagram doesn't operate any user-facing Cassandra servers, though. They run user-facing web servers that talk to Cassandra internally. I don't like the AGPL because it's unclear on this exact sort of thing, but it does seem to me like the obvious reading of "all users interacting with it remotely through a computer network" does not encompass the connection between Instagram end users and their internal Cassandra.…

Counldn't sticking a proxy in front of any AGPL software defeat its purpose then? If you don't consider transitive connections it seems pointless to me.

That's probably a question for a lawyer, but I would not be surprised at an interpretation that a proxy that just mirrors the API of the thing it proxies doesn't insulate you from license compliance, in the same way that a library that just wraps a GPL library doesn't insulate you from license compliance. The question is whether the user is interacting with the AGPL product - if you're talking to software via a proxy, you'd likely say you're interacting with it, but if you're talking to some other software that happens to use that software, are you really interacting with it?

I guess the weird case is that when I'm using the Instagram app, I wouldn't say I'm personally interacting with even the Instagram front-end servers (the way I am in a browser), I'm just interacting with the app which happens to use the servers. And that does sound like not what the license authors would like.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#160

Post by tytytytytytytyt » Wed, Mar 07, 2018, 7:03 AM UTC

teacpde wrote: post #158

the8472 wrote: post #86

Earlier quoted context omitted.

No, that would be an incremental GC working in very small time slices. A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. They are still CPU cycles you don't have to serve other requests, hence they still affect throughput. That is a simplified explanation of course, there are a lot of caveats. In my original post…

> A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. That makes sense, how about GC for single-threaded languages, e.g. Nodejs?

Just because a language is single threaded in the code you write doesn't mean it doesn't use threads behind the scenes, like for GC or other things.