Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers

I may have missed it, but in many scenarios, the reason p95 etc matters isn't that it is 5% of cases ('of course 3g users are slow') but each user may issue many requests. Ex: If serving 20 assets over a session, most users will be hit w a great p50 and bad p95.

Trouble shootong at the session-of-indiv-requests level is tough, so being able to zoom in/out is the power of correlation IDs and observability stacks (vs this kind of monitoring view afaict).

I tend to use cumulative distribution functions when comparing distributions.

Histograms also tend to be quite easy to manipulate

> Well, when it comes to performance - you can’t use the average if you don’t know the distribution.

This is frankly wrong. Performance comes in multiple flavors. Latency is one of those, and there we know that percentiles really matter (see Andrew Certain's section of this talk: https://www.youtube.com/watch?v=sKRdemSirDM&feature=youtu.be... for Amazon's experience).

But for others, like throughput and scale, you don't need to know the distribution. In fact for throughput, the only thing that really matters is the long-term mean latency. For concurrency, it's that and long-term mean arrival rate. I wrote a blog post about it a while back (http://brooker.co.za/blog/2017/12/28/mean.html).

The core point here is that all summary statistics are misleading. You need to be clear on what you care about, and making absolute statements about the mean isn't a good way to do that.

Edit: This came across a bit more confrontational than I had intended. The OP makes some good points, but I think his point about the mean is overly broad.

Heat maps (showing histogram bin count as color/intensity instead of bar height) and scatterplots (potentially of a subset of data points) are both relatively straightforward ways to visualize distributions over time.

The Circllhist algorithm is interesting, but I get an uneasy feeling from the use of a _relative error_ measure to evaluate the performance of quantile estimation. Note that other authors don't use this measure.

Dunning uses Mean Absolute Error in his latest T-digest paper: https://arxiv.org/pdf/1902.04023.pdf

Cohen uses Normalized Root-Mean-Squared Error to evaluate sampling schemes, which are equally capable of estimating latency quantiles: https://dl.acm.org/doi/abs/10.1145/3234338

The problem with Relative Error as a measure of accuracy is that it depends on the location of the distribution. The same size absolute error becomes a large relative error near zero and becomes a small relative error farther up the number line.

Another thing about this study is that only one value for the T-digest quality is tested. Of course, the T-digest quality parameter equates directly with compressed size, so it's unsurprising that T-digest's size is fixed throughout the experiment. I also suspect that the choice of data set matters quite a lot in this study. If your latency values were clustered around a small range, then the algorithms like DDSketch and Circllhist will indeed have relative error less than 5% (as they prove) but T-digest will be significantly more accurate.