Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Average Latency Is a Terrible Way To Track Website Performance

mvolo.com

Post reply on HN
Print view View original

11–20 of 31 posts

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#11

Post by mvolo »

stephencanon wrote: post #8
aetherson wrote: post #2

TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…

> Average anything is a terrible way to track anything. Came here to say exactly this. And averages are especially insidious when used for data that doesn't have a symmetric distribution, like most latencies.

hi Steve,

Author here. I think most people on HN would echo your sentinment about averages wholesale ... But I wanted to go a little deeper into selecting a better alternative for operational monitoring.

Its easy to say "averages are bad" but harder to say "use X instead", and explain why. We tried. Do you think we did it?

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#12

Post by pla3rhat3r »

taylorbuley wrote: post #4

One problem I have with this approach is that it requires you to pick a threshold after which the response is "too slow." This number can change a lot over the course of an application lifetime, and would be hard to pick objectively anyways. Median latency -- perhaps with (the smoothing-effect of) a rolling median -- would be more robust to outliers without having to resort to hardcoding of "too slow" thresholds. It…

I agree with this. It's hard to gauge what is acceptable because it really depends on the application. So many other dependencies when dealing with latency and how it effects performance.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#13

Post by mjn »

aetherson wrote: post #2

TL;DR: Average anything is a terrible way to track anything. (And median or mode are bad, too). Any single-scalar value that compresses information that is best expressed as a graph (or multiple graphs!) is immensely lossy to the point where arguably it obfuscates more than it makes clear. Back when we had to live with sort of printing-press methods of displaying information (ie, where anything that wasn't pure text…

This was one of the more interesting realizations when I was an undergraduate writing my first research paper. We were testing latency of MIDI interfaces, and after sanity checking by looking at some of the underlying data, realized that average, or even average+stddev, was obscuring a lot of stuff. For example, note-to-note consistency is a major issue in music interfaces, often more important than absolute latency, since the spacing between notes is very important to melody perception (games often have a similar issue).

Showing the full histogram isn't a full solution either, though. Not only does using the average latency obscure the issue by boiling it down to a single scalar, but the full histogram of latencies also loses the information on note-to-note consistency! That's because a latency histogram loses sequencing information, so it doesn't distinguish between the case where you had a lot of 20ms latencies in a row followed by a lot of 50ms latencies in a row, and the case where every other message oscillated between 20ms and 50ms latencies (much worse). You can try to capture some of that information by making a histogram of adjacent-latency deltas, as one attempt. Or you can capture a different view on it by plotting latency vs. time and looking for spikes (but that can obscure less-obvious trends, and is unwieldy as a data representation if you're trying to summarize a system's behavior over a period of hours).

The paper is here, though the actual numbers are 9 years old at this point, so probably not that useful: http://www.cs.hmc.edu/~bthom/res/midi_timing/publications/IC...

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#14

Post by mvolo »

pla3rhat3r wrote: post #12
taylorbuley wrote: post #4

One problem I have with this approach is that it requires you to pick a threshold after which the response is "too slow." This number can change a lot over the course of an application lifetime, and would be hard to pick objectively anyways. Median latency -- perhaps with (the smoothing-effect of) a rolling median -- would be more robust to outliers without having to resort to hardcoding of "too slow" thresholds. It…

I agree with this. It's hard to gauge what is acceptable because it really depends on the application. So many other dependencies when dealing with latency and how it effects performance.

hi, author here.

Unfortunately, you HAVE TO do it. If you do not set a threshold for what is acceptable, how do you determine whether or not your are providing an acceptable experience to your users?

No amount of aggregate metrics can help you answer this question unless you know whats acceptable, and what isnt - for each important set of URLs in your app.

I agree that its "hard" to do. In our own product (https://www.leansentry.com), we solve this problem by grouping urls, and using good defaults / making it easy to override the thresholds for users.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#15

Post by RyanZAG »

taylorbuley wrote: post #4

One problem I have with this approach is that it requires you to pick a threshold after which the response is "too slow." This number can change a lot over the course of an application lifetime, and would be hard to pick objectively anyways. Median latency -- perhaps with (the smoothing-effect of) a rolling median -- would be more robust to outliers without having to resort to hardcoding of "too slow" thresholds. It…

This actually calls for advances in analytics packages. If you could specify the threshold on a per page (or regular expression for complex requests), and have the system track and notify you of threshold exceptions, this wouldn't be too difficult to manage with some nice sliders.

Sounds like a good problem for an analytics startup to tackle.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#16

Post by taproot »

mvolo wrote: post #10
edouard1234567 wrote: post #9

I think there's something to be said about keeping some key metrics super simple so that "everybody" can understand without having to refer to a formula or arbitrarily set thresholds. I've been using 99 and 90 percentile avg performance. It captures enough information in most cases and doesn't require any explanation.

Hi edouard, I completely agree! Keeping toplevel metrics SIMPLE is the key. Of course, simple but also not misleading you into any wrong beliefs. While we liked the 95 percentile approach, we decided against it. Its still too focused on the actual response time itself, which we thought was less relevant than the number of users experiencing bad performance. I think for us the bottom line was: A) If you are having a s…

Im seeing a lot of "averages are bad" etc but I think you come closest to what I had in mind: there isnt anything inherently wrong with using simple metrics. The caveat is you just need to keep in mind and understand their limitations and where they fall down. I think a lot of people understand that using 99 or 95 percentiles and what not but just failed to lay the reasoning out.

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#17

Post by acdha »

Jabbles wrote: post #3

I thought looking at the 99th (or other) percentile was pretty standard practice?

Depends on standard - within the clued-in performance community, yes, but there are major, major companies still pushing averages and that causes a lot of people, particularly those without much stats / engineering background, to expect it everywhere.

To use one example which is prevalent throughout marketing, advertising, etc. Google Analytics reports only averages – this makes the results unreliable enough that I'm now advising people to simply pretend that field does not exist as it's completely untrustworthy. Awhile back I blogged about an example where 3 samples out of 200K threw the average off by a full order of magnitude: http://chris.improbable.org/2012/05/18/google-analytics-dece...

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#18

Post by Jabbles »

acdha wrote: post #17
Jabbles wrote: post #3

I thought looking at the 99th (or other) percentile was pretty standard practice?

Depends on standard - within the clued-in performance community, yes, but there are major, major companies still pushing averages and that causes a lot of people, particularly those without much stats / engineering background, to expect it everywhere. To use one example which is prevalent throughout marketing, advertising, etc. Google Analytics reports only averages – this makes the results unreliable enough that I'm…

Very interesting, thank you. I especially like the replies from the Google analytics team, 8 months apart, that both acknowledge the issue and say they'll fix it...

Re: Why Average Latency Is a Terrible Way To Track Website Performance

#20

Post by rm999 »

nateabele wrote: post #19

Searched the page for "standard deviation". Didn't find it. Hit the back button.

Standard deviation isn't the problem, skew is. Yes, skew will increase the standard deviation, but the heart of the issue here is how fat the right tail of the distribution is.

Standard deviation is often a useful metric, but it's at least as flawed as mean in skewed distributions because it doesn't treat either direction around the (already flawed) mean any differently.

Post reply on HN