Latency Sneaks Up on You
brooker.co.za
Latency Sneaks Up on You
1–10 of 27 posts
Re: Latency Sneaks Up on You
#2Mostly a reminder/clarification of things I knew, but a good and welcome one well-stated, because I probably sometimes forget. (I don't do performance work a lot).
But this:
> If you must use latency to measure efficiency, use mean (avg) latency. Yes, average latency
Not sure if I ever thought about it before, but after following the link[1] where OP talks more about it, they've convinced me. Definitely want mean latency at least in addition to median, not median alone.
Re: Latency Sneaks Up on You
#3Mostly agreed, and I think the point about efficiency working against latency is both important and widely ignored. And not just in software, but software process.
There's a great book called Principles of Product Development Flow. It carefully looks at the systems behind how things get built. Key to any good feedback loop is low latency. So if we want our software to get better for users over time, low latencies from idea to release are vital. But most software processes are tuned for keeping developers 100% busy (or more!), which drastically increases system latency. That latency means we get a gain in efficiency (as measured by how busy developers are) but a loss in how effective the system is (as determined by creation of user and business value).
Re: Latency Sneaks Up on You
#4Great article, and a line of reasoning that ought to be more widely known. There is a similar tradeoff between latency and utilization in hash tables, for essentially the same reason.
The phenomenon described by the author can lead to interesting social dynamics over time. The initial designer of a system understands the latency/utilization tradeoff and dimensions the system to be underutilized so as to meet latency goals. Then the system is launched and successful, so people start questioning the low utilization, and apply pressure to increase utilization in order to reduce costs. Invariably latency goes up, customers complain. Customers escalate, and projects are started to reduce latency. People screw around at the margin changing number of threads etc, but the fundamental tradeoff cannot be avoided. Nobody is happy in the end. (Been through this cycle a few times already.)
Re: Latency Sneaks Up on You
#5Seems like a trivially simple article, and I remain unconvinced of the conclusion. I think this is a confident beginner giving holistic, overly prescriptive advice. That is to say: feel free to skip and ignore.
In my experience, if you want monitoring (or measuring for performance) to provide any value what so ever, you must measure multiple different aspects of the system all at once. Percentiles, averages, load, responses, i/o, memory, etc etc.
The only time you would need a single metric would possibly be for alerting, and a good alert (IMHO) is one that triggers for impending doom, which the article states percentiles are good for. But I think alerts are outside of the scope of this article.
TLDR; Review of the article: `Duh`
Re: Latency Sneaks Up on You
#6Seems like a trivially simple article, and I remain unconvinced of the conclusion. I think this is a confident beginner giving holistic, overly prescriptive advice. That is to say: feel free to skip and ignore. In my experience, if you want monitoring (or measuring for performance) to provide any value what so ever, you must measure multiple different aspects of the system all at once. Percentiles, averages, load, re…
You characterization of Marc Brooker as a "confident beginner" is incorrect. The guy is a senior principal engineer at AWS, was the leader of EBS when I interacted with him, and has built more systems than I care to mention. The phenomenon he is describing is totally real. Of course the article is a simplification that attempts to isolate the essence of a terrifyingly complex problem.
Re: Latency Sneaks Up on You
#7Seems like a trivially simple article, and I remain unconvinced of the conclusion. I think this is a confident beginner giving holistic, overly prescriptive advice. That is to say: feel free to skip and ignore. In my experience, if you want monitoring (or measuring for performance) to provide any value what so ever, you must measure multiple different aspects of the system all at once. Percentiles, averages, load, re…
Re: Latency Sneaks Up on You
#8OK. I am stupid. I dont understand the article.
>> If you must use latency to measure efficiency, use mean (avg) latency. Yes, average latency
What is wrong with measuring latency at 99.99 percentile with a clear guideline that optimising efficiency ( in this article higher utilisation ) should not have trade off on latency?
Because latency is part of user experience. And UX comes first before anything else.
Or does it imply that there are lot of people who dont know the trade off between latency and utilisation? Because I dont know anyone who has utilisation to 1 or even 0.5 in production.
Re: Latency Sneaks Up on You
#9Mostly a reminder/clarification of things I knew, but a good and welcome one well-stated, because I probably sometimes forget. (I don't do performance work a lot). But this: > If you must use latency to measure efficiency, use mean (avg) latency. Yes, average latency Not sure if I ever thought about it before, but after following the link[1] where OP talks more about it, they've convinced me. Definitely want mean lat…
> If we're expecting 10 requests per second at peak this holiday season, we're good.
Problem is, sometimes system engineers do not know what to expect, but they still need to have a plan for this case.
Re: Latency Sneaks Up on You
#10OK. I am stupid. I dont understand the article. >> If you must use latency to measure efficiency, use mean (avg) latency. Yes, average latency What is wrong with measuring latency at 99.99 percentile with a clear guideline that optimising efficiency ( in this article higher utilisation ) should not have trade off on latency? Because latency is part of user experience. And UX comes first before anything else. Or does…
Percentiles are order statistics, they are robust and not sensitive to outliers. This is why sometimes they are very useful. And this is why they do not capture how big the remaining 0.01% of the data are.
Let's take a median, which is also an order statistics. And a sequence of latency measurements: 0.005 s, 0.010 s, 3600 s. Median latency is 0.010 s, and this number does not tell how bad latency can actually be. Mean latency is 1200.05 s, which is more indicative how bad the worst case is.
In other words, percentiles show how often a problem happens (does not happen). Mean values show the impact of the problem.