Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
1–10 of 31 posts
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#2> Well, when it comes to performance - you can’t use the average if you don’t know the distribution.
...and if you have the distribution, you no longer need the average!
Latency as experienced by the end user is dominated by the fat fail, for both technical reasons and psychological ones.
Technical ones are probably the most convincing: it is very rare to have users submit single requests and then be done. Especially in the cloudified, service-oriented stacks of today, even single requests lead to a cascade of requests inside the system. Whenever you have tens or hundreds of requests for a single user, it starts becoming very likely that they hit that fat tail at some point in their journey.
Given that latency is dominated by "outliers", looking at anything but p99 and beyond is meaningless.
What's worse is this: since most people at best look at p95 or p99, they tend to optimise for "the common case" at the cost of tail latencies! They introduce insane variance in latencies making benchmarks better, but things actually get worse for real users.
Sorry, this is a pet peeve of mine.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#3Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but not that good for analyzing them.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#4Shameless plug. Just wrote a paper about this:
https://arxiv.org/abs/2001.06561
Containing a survey of the most popular Latency Aggregation methods used in the industry (Prometheus Histograms, t-digest, HDR-Histogram, DD-sketch/histogram).
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#5Using the median in lieu of the average isn't always a good idea either. A service could completely fail to respond almost 50% of the time and you'd still get a low median. Same holds for perc95, but to a lesser extent.
The main problem is that people try to summarize their data too early. What you want is a measure for how good or bad a single datum is and only then can you summarize the end result. And usually averages aren't a bad choice at that stage.
Averages have some particularly nice properties when dealing with dependent variables, sums of variables, and when you want to minimize the distance between your estimate and the actual value. However to take advantage of that your measure actually needs to make sense. For companies the holy grail is if you can directly express how much money you make / lose because of that single datum, but failing that you'll need to find something that's at least somewhat proportional to it.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#6I honestly think most teams would be better off measuring max on the latency curve. It at least isn’t subject to the many transform errors introduced by most metric pipelines & it is easy to explain to people without getting into why 95 vs 99 vs 99.9.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#7Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…
The main objection is that it's hard to get a 3D graph of the distribution over time. But it's still worth drawing some snapshots of the distribution.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#8Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…
> Of course graphing percentiles over time is much easier because they just represent a single value.
Ridgeline plots (joyplots) are severely underutilized.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#9I honestly think most teams would be better off measuring max on the latency curve. It at least isn’t subject to the many transform errors introduced by most metric pipelines & it is easy to explain to people without getting into why 95 vs 99 vs 99.9.
Max is always infinity/timeout/highly variable isn't it?
And it doesn't tell you when you just made all your requests 1s slower.
Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers
#10I honestly think most teams would be better off measuring max on the latency curve. It at least isn’t subject to the many transform errors introduced by most metric pipelines & it is easy to explain to people without getting into why 95 vs 99 vs 99.9.
Max is always infinity/timeout/highly variable isn't it? And it doesn't tell you when you just made all your requests 1s slower.
It’s not infinite, it better be timeout but frequently isn’t and yes there is variance to it but usually (handwave) not any more variant than the tail percentile that teams choose.
I’ve had the experience multiple times where simply shifting a graph to max shows that timeouts/load shedding is t working, then teams get to the point where they are hitting timeouts way more than they thought. Only after working through those issues do you get to actually improving latency.
The upside of simplicity in the number is only overtaken by the downside when you start chasing real time constraints in systems that don’t need it.