Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Open-sourcing a 10x reduction in Apache Cassandra tail latency

engineering.instagram.com

Post reply on HN
Print view View original

91–100 of 171 posts

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#91

Post by ninkendo »

the8472 wrote: post #86
teacpde wrote: post #80

Earlier quoted context omitted.

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

No, that would be an incremental GC working in very small time slices. A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. They are still CPU cycles you don't have to serve other requests, hence they still affect throughput. That is a simplified explanation of course, there are a lot of caveats. In my original post…

What about the requests that are processed by a core that's doing a GC? Wouldn't that cause a higher P99 latency exactly as you'd expect?

Spending 2.5% of cycles on GC doesn't mean those cycles are perfectly distributed. The distribution of GC work onto cores is bound to have some cores doing more work than others, which would (I would think) manifest itself as some requests that land on an unlucky core getting more latency. Isn't it totally expected that this would cause a P99 latency spike?

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#92

Post by threeseed »

coryfoo wrote: post #33
StreamBright wrote: post #22

In a similar situation we just adjust the GC and started to use G1GC which resulted in similar numbers.

I bet that didn't take N engineers 12 months to build out, either

Cassandra uses G1GC by default.

If it was as simple as tweaking a few GC settings to get 10x improvement pretty sure Datastax would've done it by now.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#93

Post by kbenson »

zenonu wrote: post #82
viraptor wrote: post #79

Earlier quoted context omitted.

This is a valid criticism of the methodology / explanation. It's not about the results. You can agree with the positive results (and they're great! - you've done awesome work and clearly show an improvement) and still say the explanation how/why they were achieved is not great.

Exactly. Immediately from the premise of the paper, I was looking forward to a discussion on how they tried various strategies to tune their JVM/GC parameters and found nothing. The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

> The "well I guess we gotta replace this with a C++ solution" sentiment smacks of poor software engineering practice, despite the results.

On the flip side, recognizing when you are not efficiently using your time to solve a problem with the current approach and deciding to switch to another one is what I would consider a vital part of good software engineering, and also one of the hardest things to become good at without a lot of experience.

I'm not sure that's the case here, but I don't doubt it's possible they could easily have wasted more time testing and tweaking Java than it took to implement this solution. Whether that's how it would have (or possible even did to a degree) played out is an open question.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#94

Post by [deleted] »

teacpde wrote: post #80
the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

> If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. I try to understand the meaning. Is it saying the latency caused be GC is applied to all requests, not just the ones that observe 99th percentile latency?

[deleted]

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#95

Post by _ivvf »

the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect p99 latencies.

It's pretty much accepted everywhere that GCs perform terribly for databases. Modern GCs are great at handling small, very short-lived memory allocations, and that's about it. Just about any other workload and manual memory management ends up being a much better use of your time than GC tuning.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#96

Post by the8472 »

ninkendo wrote: post #91
the8472 wrote: post #86

Earlier quoted context omitted.

No, that would be an incremental GC working in very small time slices. A concurrent GC spends CPU cycles on different cores to do its work, which means it will not cause latency outliers in the threads processing the requests. They are still CPU cycles you don't have to serve other requests, hence they still affect throughput. That is a simplified explanation of course, there are a lot of caveats. In my original post…

What about the requests that are processed by a core that's doing a GC? Wouldn't that cause a higher P99 latency exactly as you'd expect? Spending 2.5% of cycles on GC doesn't mean those cycles are perfectly distributed. The distribution of GC work onto cores is bound to have some cores doing more work than others, which would (I would think) manifest itself as some requests that land on an unlucky core getting more…

Maybe if you operated your system at the saturation point, which you really don't want to do in practice. Instead you want your queues to be mostly empty. Bursts are inevitable but bursts coinciding with GC doing work hopefully is a beyond 99th percentile thing. Of course if we're speaking purely theoretical we could also assume spherical cows in a vacuum and say that requests don't burst and simply arrive in a metronome-like trickle and then the spikes evaporate too. This is basically queuing theory.

And you could also use more CPU cores than request workers, that way you will always have spare core capacity and thus your latency will not be directly impacted. That is if you really really value latency more than throughput.

Again, my main point is that throughput and latency are not the same thing. There is some relation in so far that you cannot fulfill latency promises if your throughput is insufficient and your queues start filling up. But below the saturation point it's a lot more complicated, especially in parallel systems with bursty arrivals.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#97

Post by 3uclid »

ddorian43 wrote: post #89
3uclid wrote: post #42

Unrelated: as a CS undergrad, I read this article and was immediately inspired. This is definitely the type of work I want to be doing when I graduate (infrastructure engineering). But my next thought was: where do I start?! Any advice?

Still in school ? (don't understand different grad). See: GSOC Seastar Framework https://summerofcode.withgoogle.com/organizations/6190282903...

Yeah, still in school (3rd year). I have intern experience, but it seems like these type of positions are way too advanced for me at the moment. Just unsure how to progress...

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#98

Post by nvarsj »

_ivvf wrote: post #95
the8472 wrote: post #66

> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular?

Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#99

Post by thefounder »

nvarsj wrote: post #98
_ivvf wrote: post #95

Earlier quoted context omitted.

you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…

So why do people keep building latency sensitive things in the JVM? And then they manage to get hugely popular? Cassandra is a constant struggle with the GC. I’d guess the cost of running it is at least an order of magnitude greater compared to if it had been implemented in c++ or something more sensible.

Apparently these people enjoy GC/JVM languages more than C++.

Re: Open-sourcing a 10x reduction in Apache Cassandra tail latency

#100

Post by nemothekid »

kevindong wrote: post #67
geofft wrote: post #53

Earlier quoted context omitted.

> Scylla is AGPL for the OSS version though so testing it out would not be an option without getting a commercial license first. Huh? The AGPL is not a non-commercial-use-only license. If you have proprietary software that you would like to combine with AGPL code (i.e., not interact with as a service) and is available to the general public over the Internet, and you want keep your code proprietary, sure, you may not…

The exact interpretation varies from company to company. Some companies take the strict stance of "if you use this library in any way in your application, you must open source your entire application." I've found that some libraries explicitly state that requirement within their FAQs for their community/free edition as opposed to their commercially (and paid) licensed equivalent. At the end of the day, it's not worth…

FWIW, I’m pretty sure Facebook also uses mongodb heavily (or at least through its acquisition of parse) I don’t know if they have a commercial license or not, but they aren’t strangers to the AGPL
Post reply on HN