> The graph shows that a Cassandra server instance could spend 2.5% of runtime on garbage collections instead of serving client requests. The GC overhead obviously had a big impact on our P99 latency No, this is not obvious. If you have a fully concurrent GC then spending 25 out of 1000 CPU cycles on memory management does not "obviously" have an impact on your 99th percentile latency. It would primarily impact your…
you clearly didn't read the post very closely. They said 2.5% of CPU cycles were spent on stop-the-world young generation collections, not on the sum total of all memory mangement. That means that 2.5% of the time the app is entirely stalled on just these collections. Given that stop-the-world pauses are never evenly distributed throughout time, it should be very much expected that this much GC stalling would affect…
That is not a given. And, even distribution is only part of the equation. If they are sufficiently short, then even being somewhat unevenly distributed should not have much of an impact on latency. For example, if the max length of a pause were 1ms, and 99p latency were 15ms, you'd have to be fairly unlucky to see a 33% increase in latency99 due to GC. That would entail 5 of 25 pauses happening during a 20ms period in a 1s window.
(This idea is not purely hypothetical. For example, Go's GC has very low STW periods.)
> It's pretty much accepted everywhere
Eh. Apparently everyone thinks C is the best language for cryptography and other secure but not particularly perf sensitive code. Go figure. Sometimes the wisdom of the masses is not wisdom. Best not to appeal to it during argumentation.