Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Average Latency Is a Terrible Way To Track Website Performance

mvolo.com

Post reply on HN
Print view View original

21–30 of 31 posts

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#21

Post by rm999 »

aetherson wrote: post #2

TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…

I almost completely agree with you. I often tell people that statistics is the study of compressing information in useful ways. That said, scalar statistics can be very useful if the compression is 'correct'. For example, if you have an a priori reason to believe a distribution will be gaussian (a very common situation, and an assumption that basically allowed statistics to be grow to where it is today), mean and variance will fully describe the distribution. Many other common distributions can be fully described by a small number of parameters.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#22

Post by spitfire »

Michael Abrash talked about this in his black book of graphics programming.

WHen he was writing quake, they could trade off between lighting fast graphics (40fps+, on a 486) 99% of the time with the occasional horrible slowdown to less than 5fps. vs a steady frame rate that never changed much, but wasn't terribly fast.

Turns out people notice the occasional horrible lag much more than when things perform uniformly.

When tuning a performance critical service, focus on the outliers.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#23

Post by mvolo »

nateabele wrote: post #19

Searched the page for "standard deviation". Didn't find it. Hit the back button.

hi nateabele,

Author here. Did you also search for stdev, st.dev, variance? Just kidding.

The post is not about averages. Its about selecting the right metric to track website performance. Standard deviation would surely qualify the avg. latency a bit, but it would still be a pretty lame alternative to using a better toplevel metric like Apdex.

Best, Mike

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#24

Post by bluesmoon »

Posted this comment on the article, but thought it would be useful here as well:

Good points on why average latency is a bad metric, and while the idea behind Apdex was good, it never ended up being the right measure. The Apdex score still depends on a HiPPO (Highest Paid Person''s Opinion) to determine what T should be, and this can change over time.

At SOASTA (and previously at LogNormal), we borrowed the concept of LD50 (the median lethal dose) from biology. The LD50 value has the property of adapting to what your audience thinks rather than what your HIPPO thinks is a good experience.

We described the method at the Velocity conferences (Santa Clara and London) last year, and wrote it up in a blog post here: http://www.lognormal.com/blog/2012/10/03/the-3.5s-dash-for-a...

Hope you find it interesting.

I should also mention that it's useful to apply some kind of smoothing to timeseries data (like latency over time). Holt-Winters double-exponential smoothing is particularly good at this. What it does is smooth out temporary glitches and show you when things turn unexpectedly bad. If you've ever received a page and said, "Oh yeah, that one... that goes away in 3 seconds. Happens every day.", then you'll find this useful. H-W D-E smoothing only shows you the ones that don't go away after 3 seconds.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#25

Post by pla3rhat3r »

mvolo wrote: post #14
pla3rhat3r wrote: post #12

Earlier quoted context omitted.

I agree with this. It's hard to gauge what is acceptable because it really depends on the application. So many other dependencies when dealing with latency and how it effects performance.

hi, author here. Unfortunately, you HAVE TO do it. If you do not set a threshold for what is acceptable, how do you determine whether or not your are providing an acceptable experience to your users? No amount of aggregate metrics can help you answer this question unless you know whats acceptable, and what isnt - for each important set of URLs in your app. I agree that its "hard" to do. In our own product ( https://w…

There so many moving parts that this could either be a great tool to analyze data or it could open up a can of worms which could lead down the road to network re-architecture. If this takes into account best effort or SLA based ISP, Network topology, QoS, Packet Prioritization, etc then I think it could be a useful tool. Without it it's just a tool that spits out pretty pictures. If your main selling point is data then it has to be more than just what latency can show.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#26

Post by chetanahuja »

mvolo wrote: post #11
stephencanon wrote: post #8

Earlier quoted context omitted.

> Average anything is a terrible way to track anything. Came here to say exactly this. And averages are especially insidious when used for data that doesn't have a symmetric distribution, like most latencies.

hi Steve, Author here. I think most people on HN would echo your sentinment about averages wholesale ... But I wanted to go a little deeper into selecting a better alternative for operational monitoring. Its easy to say "averages are bad" but harder to say "use X instead", and explain why. We tried. Do you think we did it?

Well the title seems a bit childish (since obviously everybody on HN knows it's a terrible idea.) Why don't you change to post title to more appropriately reflect what you were trying to propose as an alternative.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#27

Post by mvolo »

pla3rhat3r wrote: post #25
mvolo wrote: post #14

Earlier quoted context omitted.

hi, author here. Unfortunately, you HAVE TO do it. If you do not set a threshold for what is acceptable, how do you determine whether or not your are providing an acceptable experience to your users? No amount of aggregate metrics can help you answer this question unless you know whats acceptable, and what isnt - for each important set of URLs in your app. I agree that its "hard" to do. In our own product ( https://w…

There so many moving parts that this could either be a great tool to analyze data or it could open up a can of worms which could lead down the road to network re-architecture. If this takes into account best effort or SLA based ISP, Network topology, QoS, Packet Prioritization, etc then I think it could be a useful tool. Without it it's just a tool that spits out pretty pictures. If your main selling point is data th…

hi there,

The post is about selecting a top level metric for monitoring website performance. One a problem is indicated, you would definitely need to drill in to figure out what part of your app is affected, when, and what caused it.

LeanSentry (our own application monitoring product, https://www.leansentry.com) does this. However, describing this was outside the scope of my post (but you can see the demo of it on the website).

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#29

Post by jamesaguilar »

spitfire wrote: post #22

Michael Abrash talked about this in his black book of graphics programming. WHen he was writing quake, they could trade off between lighting fast graphics (40fps+, on a 486) 99% of the time with the occasional horrible slowdown to less than 5fps. vs a steady frame rate that never changed much, but wasn't terribly fast. Turns out people notice the occasional horrible lag much more than when things perform uniformly. W…

I think you should not ignore either. By default, think about 99%ile and 50%ile when tuning and optimizing. Depending on the context (e.g. games), even 99%ile might not be enough, or you might want to think about 99%ile of what? Frames? Scenes? Seconds of gameplay?

Also, back to the topic of the article at hand, I hope that their "T" is not really two seconds. That is already way too slow for most web purposes.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#30

Post by jacques_chester »

stephencanon wrote: post #8
aetherson wrote: post #2

TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…

> Average anything is a terrible way to track anything. Came here to say exactly this. And averages are especially insidious when used for data that doesn't have a symmetric distribution, like most latencies.

Additional standard statistics like mode, median, quartiles etc are really useful.

And you can always throw things into gnuplot to get a quick, exploratory look at things. It will at least give you sense of whether you're looking at a normal distribution, something skewed, multi-modal distributions etc etc.

Post reply on HN