Thank you for writing this article, there's a deep and powerful insight illustrated here: An observer using the system experiences different statistics from the system operator. By extension, taking an average of observer experiences leads to different conclusions from taking an average of system performance. One must not confuse the two when designing systems.
Alice is impatient
31–40 of 46 posts
Re: Alice is impatient
#32Earlier quoted context omitted.
Perhaps an easier to intuit version of it is how full airliners are. An airline might report that their flights are on average 60% full, and that might be completely absolutely 100% true. But that's not what passengers experience. If we assume (for convenience) that a plane holds 100 people, when the plane is 20% full then 20 passengers experience that, but when the plane is 100% full then 100 passengers experience t…
I think you got the `900*9` wrong if you talk about experienced downtime. If you calculate discrete minutes (900*4,5+100*1)/1000 = 4,15 min (Unless you manage to inform the user since how long the website has been down already.) This could be made more accurate if we calculate it over seconds, which would drive the experienced downtime even lower!
It's 100*10*4.5 inside, you're summing 100*9+100*8+...+100*1.
So 4.6min, i.e. 4:36
Re: Alice is impatient
#33Considering other metrics then p99 for user impact is unwise. All users will at some point experience a By focusing on the tail and optimizing worst cases you help users more than by improving your median latency.
If your frontend fires hundreds of requests (which isn't uncommon) then the p99 is merely what most users will experience. Ideally you want cumulative distribution chart that goes up to the max.
And then that's just for the requests you measure. If something takes too long the user might do something that cancels the requests which means the backend never completes its response and won't get the time-to-response sample, so you need to account dropped requests too.
Re: Alice is impatient
#34This is the same calculation behind the observation "you spend much longer in front of red traffic lights than green ones".
It's an interesting observation, but it's playing games with the meaning of "mean latency" and I'm not sure this is a very helpful way to look at requests to a web service - slightly slowing the fastest responses to requests would improve your time-weighted mean latency.
It's a better metric for looking at outages - instantaneous outages don't affect anyone, and time-weighting correctly discards them. On the other hand, average outage length is a very suspect and gameable metric unless accompanied by uptime %.
Re: Alice is impatient
#35There is a branch of math dedicated to (among other things) truthfully estimating the waiting time, called queueing theory. I wonder why it wasn't mentioned in the article.
Re: Alice is impatient
#36Considering other metrics then p99 for user impact is unwise. All users will at some point experience a By focusing on the tail and optimizing worst cases you help users more than by improving your median latency.
If your frontend fires hundreds of requests (which isn't uncommon) then the p99 is merely what most users will experience. Ideally you want cumulative distribution chart that goes up to the max. And then that's just for the requests you measure. If something takes too long the user might do something that cancels the requests which means the backend never completes its response and won't get the time-to-response samp…
This is only true if your latency distribution is fully random, which is rarely the case. More often than not, it's the same small group of users hitting most of the p99 because their accounts are simply more resource intensive.
Re: Alice is impatient
#37This feels similar to how nuclear power is perceived, contrasting deaths per TWh and the long tail effect of a rare but serious accident.
Re: Alice is impatient
#38> Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s.
There are also plenty of situations where a service can have a bimodal performance distribution and the impact of that can fall on certain users disproportionately.
Imagine a retail website that serves images from a global CDN, with cache misses pulled from a server in the EU. Users who visit our homepage, or look at our bestselling products, get a cache hit from the CDN node close to them, in 50ms. But users who look at our long-tail products get a cache miss - and if they're not near Europe, they'll get a noticeable delay.
Hence our mean image load time is 100ms - but a customer browsing an obscure product category for their location can experience markedly worse performance. If Alice is the only person in Costa Rica looking at ski equipment in June, she's going to get a lot of cache misses.
Re: Alice is impatient
#39I don’t remember any service I used in the last couple of years, where I thought to myself: this service is really fast and responsive. Great experience.
Quite the contrary: feels like everything got worse. Sometimes painfully slow, buggy and unreliable.
Re: Alice is impatient
#40I've grown to dislike the typical tail measurements completely. What I usually look at these days is what share of unique users experience an "unacceptable experience" over a measurement period instead. I find it much more inquisitive and visceral, to the extent that p99 now boggles my mind. 2N would be dreadful as an availability figure, yet for UX it's treated very different. So much so that my measurements corrobo…
[flagged]