Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How percentile approximation works and why it's more useful than averages

blog.timescale.com

Post reply on HN
Print view View original

21–30 of 173 posts

Re: How percentile approximation works and why it's more useful than averages

#21

Post by contravariant »

rahimnathwani wrote: post #4

For some things, you can't even sensibly measure the mean. For example, if you're measuring the mean response time for a service, a single failure/timeout makes the mean response time infinite (because 100 years from now the response still hasn't been received). "Why Averages Suck and Percentiles are Great": https://www.dynatrace.com/news/blog/why-averages-suck-and-pe...

It's indeed always worth pointing that a mean may or may not exist. Same with variances/standard-deviation.

The central limit theorem seems to have given people the slightly wrong idea that things will always average out eventually.

Re: How percentile approximation works and why it's more useful than averages

#22

Post by doctorsher »

madars wrote: post #8

Good opportunity to plug https://en.wikipedia.org/wiki/Anscombe%27s_quartet : if you don't know much about the underlying distribution, simple statistics don't describe it well. From Wikipedia description: Anscombe's quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x…

This is excellent information, thank you for posting this! I was not familiar with this example previously, but it is a perfect example of summary statistics not capturing certain distributions well. It's very approachable, even if you had to limit the discussion to mean and variance alone. Bookmarked, and much appreciated.

Re: How percentile approximation works and why it's more useful than averages

#23

Post by djk447 »

cyral wrote: post #13
bqe wrote: post #3

Awhile ago I wrote a Python library called LiveStats[1] that computed any percentile for any amount of data using a fixed amount of memory per percentile. It uses an algorithm I found in an old paper[2] called P^2. It uses a polynomial to find good approximations. The reason I made this was an old Amazon interview question. The question was basically, "Find the median of a huge data set without sorting it," and the "…

There are some newer data structures that take this to the next level such as T-Digest[1], which remains extremely accurate even when determining percentiles at the very tail end (like 99.999%) [1]: https://arxiv.org/pdf/1902.04023.pdf / https://github.com/tdunning/t-digest

NB: Post author here.

Yeah, that was one of the reasons we chose it as one of the ones to implement, seemed like that was a really interesting tradeoff, we also used uddsketch[1] which provides relative error guarantees, which is pretty nifty. We thought they provided different enough tradeoffs that we wanted to implement both.

[1]: https://arxiv.org/abs/1908.10693

Re: How percentile approximation works and why it's more useful than averages

#24

Post by pdpi »

madars wrote: post #8

Good opportunity to plug https://en.wikipedia.org/wiki/Anscombe%27s_quartet : if you don't know much about the underlying distribution, simple statistics don't describe it well. From Wikipedia description: Anscombe's quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x…

There's a fun paper by Autodesk where they make datasets that look whatever way you want them to.

https://www.autodesk.com/research/publications/same-stats-di...

Re: How percentile approximation works and why it's more useful than averages

#25

Post by djk447 »

pdpi wrote: post #24
madars wrote: post #8

Good opportunity to plug https://en.wikipedia.org/wiki/Anscombe%27s_quartet : if you don't know much about the underlying distribution, simple statistics don't describe it well. From Wikipedia description: Anscombe's quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x…

There's a fun paper by Autodesk where they make datasets that look whatever way you want them to. https://www.autodesk.com/research/publications/same-stats-di...

NB: Post author here.

This is great! So fun...will have to use in the future...

Re: How percentile approximation works and why it's more useful than averages

#26

Post by esyir »

waynecochran wrote: post #12
djk447 wrote: post #9

Earlier quoted context omitted.

NB: Post author here. Std deviation definitely helps a lot, still often not as good as percentiles, was actually thinking about adding some of that in the post but it was already getting so long. It's funny how things you think are simple sometimes take the most effort to explain, definitely found that on this one.

Yeah -- std deviation has a similar problem to the mean in that it doesn't give you a full picture unless the distribution is close to normal / gaussian.

Pretty much why summary statistics often give the IQR, which gives some idea to the skew and shape of the distribution as well.

Unfortunately, BD and marketing just want a single number to show that the value is bigger and hate anything more complicated than a barchart.

Re: How percentile approximation works and why it's more useful than averages

#27

Post by varelaz »

esyir wrote: post #26
waynecochran wrote: post #12

Earlier quoted context omitted.

Yeah -- std deviation has a similar problem to the mean in that it doesn't give you a full picture unless the distribution is close to normal / gaussian.

Pretty much why summary statistics often give the IQR, which gives some idea to the skew and shape of the distribution as well. Unfortunately, BD and marketing just want a single number to show that the value is bigger and hate anything more complicated than a barchart.

Barchart is basically your percentiles (just more of them) so why not show it? Bars and whiskers could be more complicated for them but still the same sort of data

Re: How percentile approximation works and why it's more useful than averages

#29

Post by [deleted] »

madars wrote: post #8

Good opportunity to plug https://en.wikipedia.org/wiki/Anscombe%27s_quartet : if you don't know much about the underlying distribution, simple statistics don't describe it well. From Wikipedia description: Anscombe's quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x…

[deleted]

Re: How percentile approximation works and why it's more useful than averages

#30

Post by bluesmoon »

This is exactly the algorithm we developed at LogNormal (now part of Akamai) 10 years ago for doing fast, low-memory percentiles on large datasets.

It's implemented in this Node library: https://github.com/bluesmoon/node-faststats

Side note: I wish everyone would stop using the term Average to refer to the Arithmetic mean. "Average" just means some statistic used to summarize a dataset. It could be the Arithmetic Mean, Median, Mode(s), Geometric Mean, Harmonic Mean, or any of a bunch of other statistics. We're stuck with AVG because that's the function used by early databases and Lotus 123.

Post reply on HN