Why Average Latency Is a Terrible Way To Track Website Performance
1–10 of 31 posts
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#2TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear.
Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text was very difficult to display), mean/median/mode numbers were a necessary evil. But if you're looking at a computer screen, there's really no reason to subject yourself to an abstraction that throws out 90% of your data.
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#3I thought looking at the 99th (or other) percentile was pretty standard practice?
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#4One problem I have with this approach is that it requires you to pick a threshold after which the response is "too slow." This number can change a lot over the course of an application lifetime, and would be hard to pick objectively anyways.
Median latency -- perhaps with (the smoothing-effect of) a rolling median -- would be more robust to outliers without having to resort to hardcoding of "too slow" thresholds. It would still require the human to connect the dots (e.g. median latency of >200 is "too slow") but it's an improvement on mere average response time for reasons noted.
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#5TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…
I think The Tech Report has my favorite benchmark graphs. They sort the data points by latency so you can intuitively see the distribution of your samples. e.g. http://techreport.com/review/24022/does-the-radeon-hd-7950-s...
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#6TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…
Hi, author here.
I am in complete agreement. Unfortunately, a lot of monitoring and APM tools still lead with average response time as one of the toplevel metrics. And a lot of people still make incorrect assumptions based on it.
Although, the percentile on average latency is not great either. I try to make the case for using a metric that counts acceptable experiences vs. their latency value, e.g. the Apdex index or our derived sat score.
Best, Mike
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#7I thought looking at the 99th (or other) percentile was pretty standard practice?
hi Jabbles,
Author here. The 99 or 95 percentile is a much better metric! We also make the case for the industry standard Apdex or our derived metric, sat score. These are becoming more and more in use by APM tools like us or New Relic.
Unfortunately, many existing tools and people who use them still look at latency aggregates and often make incorrect assumptions.
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#8TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…
> Average anything is a terrible way to track anything.
Came here to say exactly this. And averages are especially insidious when used for data that doesn't have a symmetric distribution, like most latencies.
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#9I think there's something to be said about keeping some key metrics super simple so that "everybody" can understand without having to refer to a formula or arbitrarily set thresholds. I've been using 99 and 90 percentile avg performance. It captures enough information in most cases and doesn't require any explanation.
Re: Why Average Latency Is a Terrible Way To Track Website Performance
#10I think there's something to be said about keeping some key metrics super simple so that "everybody" can understand without having to refer to a formula or arbitrarily set thresholds. I've been using 99 and 90 percentile avg performance. It captures enough information in most cases and doesn't require any explanation.
Hi edouard,
I completely agree! Keeping toplevel metrics SIMPLE is the key. Of course, simple but also not misleading you into any wrong beliefs.
While we liked the 95 percentile approach, we decided against it. Its still too focused on the actual response time itself, which we thought was less relevant than the number of users experiencing bad performance.
I think for us the bottom line was:
A) If you are having a site-wide performance issues, 95% percentile is a good metric.
B) However, if you have more isolated issues (we find this happens more often to more mature sites), satisfaction score is better.
Best, Mike