> Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. There are also plenty of situations where a service can have a bimodal performance distribution and the impact of that can fall on certain users disproportionately. Imagine a retail website that serves images from a global CDN, with cache misses pulled from a server…
So what's the solution here? Keep the cache artificially warm for even obscure routes?
> Alice says your service is slow. You tell Alice that the mean request to your service completes in 100ms, but Alice says that her mean wait time is 1s. There are also plenty of situations where a service can have a bimodal performance distribution and the impact of that can fall on certain users disproportionately. Imagine a retail website that serves images from a global CDN, with cache misses pulled from a server…
So what's the solution here? Keep the cache artificially warm for even obscure routes?
There may not be a solution in every case, but it's a reminder that dashboards & metrics are no replacement for actually talking to your users. Metrics are at best a proxy for user experience, don't let them be the tail that wags the dog.
Like the story Bezos told of his execs claiming call wait times were under 1 minute, so he called the service line from the conference room on the spot and made everyone sit there for 10 minutes waiting to get thru..
My understanding is a lot of the probability puzzles in Allen Downey's Probably Overthinking It [1] also boil down to similar selection effects (the inspection paradox is definitely in there). There is a lot of cool stuff in that book (and his blog of the same name).
I don’t remember any service I used in the last couple of years, where I thought to myself: this service is really fast and responsive. Great experience. Quite the contrary: feels like everything got worse. Sometimes painfully slow, buggy and unreliable.
Try the McMaster-Carr website sometime. It's incredible. Expensive, but incredible.
So what's the solution here? Keep the cache artificially warm for even obscure routes?
There may not be a solution in every case, but it's a reminder that dashboards & metrics are no replacement for actually talking to your users. Metrics are at best a proxy for user experience, don't let them be the tail that wags the dog. Like the story Bezos told of his execs claiming call wait times were under 1 minute, so he called the service line from the conference room on the spot and made everyone sit there f…
Ok, that's a fun anecdote and I agree it has real world value to think that way. But it doesn't answer my question - this thread is full of people pointing out problems but nobody offering solutions. So I was asking, specifically in the case of an app with a very long tail of cache misses, what's the solution? Do you have to keep potentially millions of routes artificially warm?
If your frontend fires hundreds of requests (which isn't uncommon) then the p99 is merely what most users will experience. Ideally you want cumulative distribution chart that goes up to the max. And then that's just for the requests you measure. If something takes too long the user might do something that cancels the requests which means the backend never completes its response and won't get the time-to-response samp…
This is only true if your latency distribution is fully random, which is rarely the case. More often than not, it's the same small group of users hitting most of the p99 because their accounts are simply more resource intensive.
Depending on how the system distributes work such users can interfere with random with requests from other users through shared resources, so to that cohort these will look like a random latency distribution.