Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How percentile approximation works and why it's more useful than averages

blog.timescale.com

Post reply on HN
Print view View original

91–100 of 173 posts

Re: How percentile approximation works and why it's more useful than averages

#91

Post by maxnoe »

contravariant wrote: post #21
rahimnathwani wrote: post #4

For some things, you can't even sensibly measure the mean. For example, if you're measuring the mean response time for a service, a single failure/timeout makes the mean response time infinite (because 100 years from now the response still hasn't been received). "Why Averages Suck and Percentiles are Great": https://www.dynatrace.com/news/blog/why-averages-suck-and-pe...

It's indeed always worth pointing that a mean may or may not exist. Same with variances/standard-deviation. The central limit theorem seems to have given people the slightly wrong idea that things will always average out eventually.

Coughs Cauchy

Re: How percentile approximation works and why it's more useful than averages

#93

Post by robotresearcher »

ilikebits wrote: post #71
Izikiel43 wrote: post #52

Earlier quoted context omitted.

> The question was basically, "Find the median of a huge data set without sorting it," Isn't this done using a min heap and a max heap in conjuction?

The real constraint here is probably "find the median of a huge data set without holding the entire data set in memory".

'Estimate the median of an arbitrary sized data set using a constant amount of memory'.

Re: How percentile approximation works and why it's more useful than averages

#94

Post by bluesmoon »

JackFr wrote: post #45
bluesmoon wrote: post #30

This is exactly the algorithm we developed at LogNormal (now part of Akamai) 10 years ago for doing fast, low-memory percentiles on large datasets. It's implemented in this Node library: https://github.com/bluesmoon/node-faststats Side note: I wish everyone would stop using the term Average to refer to the Arithmetic mean. "Average" just means some statistic used to summarize a dataset. It could be the Arithmetic Mea…

No we’re stuck with it because average was used colloquially for arithmetic mean for decades. I wish people would stop bad-mouthing the arithmetic mean. If you have to convey information about a distribution and you’ve got only one number to do it, the arithmetic mean is for you.

Yes, Lotus 123 came out 38 years ago :)

Re: How percentile approximation works and why it's more useful than averages

#95

Post by hikerclimber1 »

bqe wrote: post #3

Awhile ago I wrote a Python library called LiveStats[1] that computed any percentile for any amount of data using a fixed amount of memory per percentile. It uses an algorithm I found in an old paper[2] called P^2. It uses a polynomial to find good approximations. The reason I made this was an old Amazon interview question. The question was basically, "Find the median of a huge data set without sorting it," and the "…

Smoking cannabis is also bad for you. It’s just a ploy by governments to get you to smoke so you die earlier.

Re: How percentile approximation works and why it's more useful than averages

#96

Post by exolymph »

hnuser123456 wrote: post #38

Gamers have an intuitive sense of this. Your average framerate can be arbitrarily high, but if you have a big stutter every second between the smooth moments, then a lower but more consistent framerate may be preferable, typically expressed as the 1% and 0.1% slowest frames, which at a relatively typical 100fps, represents the slowest frame every second and every 10 seconds.

"Slow is smooth and smooth is fast"

Re: How percentile approximation works and why it's more useful than averages

#97

Post by djk447 »

lewispb wrote: post #92

Side note, but I love the animations, code snippet design and typography in this blog post. Will think about how I can improve my own blog with these ideas.

Thank you! Huge shout out to Shane, Jacob and others on our team who helped with the graphics / design elements!

Re: How percentile approximation works and why it's more useful than averages

#99

Post by cyral »

jeffbee wrote: post #80
WireBaron wrote: post #48

Earlier quoted context omitted.

The UDDSketch (default) implementation will allow rolling percentiles, though we still need a bit of work on our end to support it. There isn't a way to do this with TDigest however.

Sure there is. You simply maintain N phases of digests, and every T time you evict a phase and recompute the summary (because T-digests are easily merged).

This is what I do, it's not a true rolling digest but it works well enough for my purposes.

Re: How percentile approximation works and why it's more useful than averages

#100

Post by michaelmdresser »

Looking particularly at latency measurements, I found the "How NOT to Measure Latency" [1] talk very illuminating. It goes quite deep into discussing how percentiles can be used and abused for measurement.

[1]: https://www.infoq.com/presentations/latency-response-time/

Post reply on HN