How percentile approximation works and why it's more useful than averages • Hacker News Board

Re: How percentile approximation works and why it's more useful than averages

#101

Post by convolvatron » Tue, Sep 14, 2021, 8:37 PM UTC

mherdeg wrote: post #58

I've skimmed some of the literature here when I've spent time trying to help people with their bucket boundaries for Prometheus-style instrumentation of things denominated in "seconds", such as processing time and freshness. My use case is a little different from what's described here or in a lot of the literature. Some of the differences: (1) You have to pre-decide on bucket values, often hardcoded or stored in code…

look at Greenwald-Khanna (and the followon work). It adapts the bucket size to minimize the total error (and the number of buckets is proportional to log-epsilon I think).

with all the competition and 'innovation', you would think the data dogs of this world would grow up beyond time series.

Re: How percentile approximation works and why it's more useful than averages

#102

Post by benlivengood » Tue, Sep 14, 2021, 8:41 PM UTC

tunesmith wrote: post #56

djk447 wrote: post #18

Earlier quoted context omitted.

NB: Post author here. I found it surprisingly difficult to explain well. Took a lot of passes and a lot more words than I was expecting. It seems like such a simple concept. I thought the post was gonna be the shortest of my recent ones, and then after really explaining it and getting lots of edits and rewriting, it was 7000 words and ...whoops! But I guess it's what I needed to explain it well (hope you thought so a…

The article hit the pocket pretty exactly for how I felt I needed to explain it. (Actually I had the same experience where I thought I would be able to go over it quickly and then I felt like I was getting mired.) The graphs look great too - I've been able to explore it pretty well with JupyterLab but I can't export that into something super-interactive that is read-only. I've thought about creating an idyll page out…

> I think the weirdly-shaped long-tail graphs we come across are just sums of more naturally-distributed response times, for different types of responses. Another reason to limit variation I think.

The best method I've found for digging into latency is profiling RPC traces to see where time is spent. This at least separates the code and parameters that have simple latency distributions from those that don't. Some distributions will be very data-dependent (SQL queries).

Re: How percentile approximation works and why it's more useful than averages

#103

Post by a-dub » Tue, Sep 14, 2021, 8:55 PM UTC

abnry wrote: post #81

One way to think about why we tend to use averages instead of medians is that it is related to a really deep theorem in probability: The Central Limit Theorem. But I think we can twist our heads and see in a way that this is backwards. Mathematically, the mean is much easier to work with because it is linear and we can do algebra with it. That's how we got the Central Limit Theorem. Percentiles and the median, except…

i think of it as: if the data is gaussian, use a mean, otherwise go non-parametric (medians/percentiles).

or put another way, if you can't model it, you're going to have to sort, or estimate a sort, because that's all that's really left to do.

this shows up in things from estimating centers with means/percentiles to doing statistical tests with things like the wilcoxon tests.

Re: How percentile approximation works and why it's more useful than averages

#104

Post by yongjik » Tue, Sep 14, 2021, 9:05 PM UTC

pachico wrote: post #46

Surprisingly, many software engineers I know never used percentiles and keep using mean average. True story.

Bah, I'll be happy if I could even get correct averages. I see pipelines getting value X1 from a server that served 100 requests, another value X2 from a server that served one request, and then it returns (X1+X2)/2.

Re: How percentile approximation works and why it's more useful than averages

#105

Post by Lightbody » Tue, Sep 14, 2021, 9:11 PM UTC

Whenever this topic comes up, I always encourage folks to watch this 2011 classic 15m talk at the Velocity conference by John Rauser:

https://www.youtube.com/watch?v=coNDCIMH8bk

Re: How percentile approximation works and why it's more useful than averages

#106

Post by fwip » Tue, Sep 14, 2021, 9:15 PM UTC

varelaz wrote: post #27

esyir wrote: post #26

Earlier quoted context omitted.

Pretty much why summary statistics often give the IQR, which gives some idea to the skew and shape of the distribution as well. Unfortunately, BD and marketing just want a single number to show that the value is bigger and hate anything more complicated than a barchart.

Barchart is basically your percentiles (just more of them) so why not show it? Bars and whiskers could be more complicated for them but still the same sort of data

Barcharts across categorical data :P

That is, the first bar is "Our Number" and the second bar is "Competitor's number."

Re: How percentile approximation works and why it's more useful than averages

#107

Post by lanstin » Tue, Sep 14, 2021, 9:32 PM UTC

5faulker wrote: post #61

Izikiel43 wrote: post #52

Earlier quoted context omitted.

> The question was basically, "Find the median of a huge data set without sorting it," Isn't this done using a min heap and a max heap in conjuction?

It's funny that this is often left out from a data & algorithm class.

Because unlike many dynamic programming algorithms, it is something anyone running a large system will need.

Re: How percentile approximation works and why it's more useful than averages

#108

Post by lanstin » Tue, Sep 14, 2021, 9:36 PM UTC

a-dub wrote: post #103

abnry wrote: post #81

One way to think about why we tend to use averages instead of medians is that it is related to a really deep theorem in probability: The Central Limit Theorem. But I think we can twist our heads and see in a way that this is backwards. Mathematically, the mean is much easier to work with because it is linear and we can do algebra with it. That's how we got the Central Limit Theorem. Percentiles and the median, except…

i think of it as: if the data is gaussian, use a mean, otherwise go non-parametric (medians/percentiles). or put another way, if you can't model it, you're going to have to sort, or estimate a sort, because that's all that's really left to do. this shows up in things from estimating centers with means/percentiles to doing statistical tests with things like the wilcoxon tests.

Assume up front none of your measured latencies from a software networked system will be Gaussian, or you will die a painful death . Even ping times over the internet have no mean. The only good thing about means is you can combine them easily, but since they are probably a mathematical fiction, combining them is even worse. Use T-Digest or one of the other algorithms being highlighted here.

Re: How percentile approximation works and why it's more useful than averages

#109

Post by lanstin » Tue, Sep 14, 2021, 9:37 PM UTC

michaelmdresser wrote: post #100

Looking particularly at latency measurements, I found the "How NOT to Measure Latency" [1] talk very illuminating. It goes quite deep into discussing how percentiles can be used and abused for measurement. [1]: https://www.infoq.com/presentations/latency-response-time/

I watch this video once a year and send it to my co-workers whenever averages or medians shows up in a graph for public consumption.

Re: How percentile approximation works and why it's more useful than averages

#110

Post by airstrike » Tue, Sep 14, 2021, 9:43 PM UTC

Sorry for the minor nitpick, but I find it a bit unusual (disappointing?) that there's an image of a candlestick chart at the very top, but the article only uses API response times as examples...