Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

schneems.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#11

Post by kasey_junk »

TeMPOraL wrote: post #8
imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

> Of course graphing percentiles over time is much easier because they just represent a single value. Ridgeline plots (joyplots) are severely underutilized.

I generally love joyplots but haven’t seen a great use of them showing latency distribution over time.

Do you know of any examples?

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#12

Post by mncharity »

TeMPOraL wrote: post #8
imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

> Of course graphing percentiles over time is much easier because they just represent a single value. Ridgeline plots (joyplots) are severely underutilized.

R https://cran.r-project.org/web/packages/ggridges/vignettes/i... ; d3 https://observablehq.com/@d3/ridgeline-plot ; py https://github.com/sbebo/joypy .

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#13

Post by alexhutcheson »

contravariant wrote: post #5

Using the median in lieu of the average isn't always a good idea either. A service could completely fail to respond almost 50% of the time and you'd still get a low median. Same holds for perc95, but to a lesser extent. The main problem is that people try to summarize their data too early. What you want is a measure for how good or bad a single datum is and only then can you summarize the end result. And usually aver…

My normal approach is to measure and set up separate alerts for "error rate" and possibly "timeout rate". You definitely want to know about those, but "mean latency" mixes those metrics with the latency metrics for successful requests, which makes it less sensitive to changes in either one.

In general I agree that averages aren't always bad. One additional advantage is that it's often possible to generate robust confidence intervals for averages, but it's often not valid to generate CIs for medians/percentiles without introducing other probably flawed assumptions.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#14

Post by contravariant »

alexhutcheson wrote: post #13
contravariant wrote: post #5

Using the median in lieu of the average isn't always a good idea either. A service could completely fail to respond almost 50% of the time and you'd still get a low median. Same holds for perc95, but to a lesser extent. The main problem is that people try to summarize their data too early. What you want is a measure for how good or bad a single datum is and only then can you summarize the end result. And usually aver…

My normal approach is to measure and set up separate alerts for "error rate" and possibly "timeout rate". You definitely want to know about those, but "mean latency" mixes those metrics with the latency metrics for successful requests, which makes it less sensitive to changes in either one. In general I agree that averages aren't always bad. One additional advantage is that it's often possible to generate robust conf…

Error rate and timeout rate are good examples of an average that measures the thing you're actually interested in.

The whole point is that trying to average the latency and only then try to figure out what it means is backwards.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#15

Post by pdpi »

imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

Histograms, too, bin values and thus show only a few values instead of the whole distribution.

In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#16

Post by OliverJones »

imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

Yes, absolutely, show the distribution whenever you can.

But consider production alerting purposes. For example, consider database user-lookup time. Use the 95th percentile, or even the maximum value to compare with an alert threshold. In many (most?) real-world cases system trouble shows up as a few lookup-time outliers. If you only look at averages, or even medians, you miss those outliers.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#17

Post by djspoons »

pdpi wrote: post #15
imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

Histograms, too, bin values and thus show only a few values instead of the whole distribution. In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.

Well, yeah, there's not much difference between a histogram and hundreds-of-percentiles. But there's a huge difference between a histogram and just a single percentile (whether it's p50, p95, p99, or whatever).

If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really important. Most of the time there are discrete behaviors/factors that are driving performance: timeouts, cache hits/misses, one overloaded host, a canary. The shape of the histogram will help you see those in a way that a single percentile can't.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#18

Post by kqr »

imtringued wrote: post #3

Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but no…

Heatmaps sorta-kinda solve the problem of plotting histograms over time, except of course it's harder to detect differences in brightness than size.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#19

Post by kqr »

lonelappde wrote: post #9
kasey_junk wrote: post #6

I honestly think most teams would be better off measuring max on the latency curve. It at least isn’t subject to the many transform errors introduced by most metric pipelines & it is easy to explain to people without getting into why 95 vs 99 vs 99.9.

Max is always infinity/timeout/highly variable isn't it? And it doesn't tell you when you just made all your requests 1s slower.

It is highly variable, and in my experience aiming to reduce that variability is one of the most reliable ways to increase total system performance, which is frequently dominated by max latencies.

In other words: max latency often indicates some degenerate case (assignable causes) that you want to work out of your system to improve its performance.

Re: Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

#20

Post by djspoons »

djspoons wrote: post #17
pdpi wrote: post #15

Earlier quoted context omitted.

Histograms, too, bin values and thus show only a few values instead of the whole distribution. In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.

Well, yeah, there's not much difference between a histogram and hundreds-of-percentiles. But there's a huge difference between a histogram and just a single percentile (whether it's p50, p95, p99, or whatever). If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really impor…

Actually one of my coworkers gave a talk at Facebook Performance summit on this: https://www.youtube.com/watch?v=EG7Zhd6gLiw (Disclaimer: brief product pitch in the first minute as part of the speaker intro)
Post reply on HN