Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

%CPU utilization is a lie

brendanlong.com

Post reply on HN
Print view View original

131–140 of 176 posts

Re: %CPU utilization is a lie

#131

Post by fluoridation »

PathOfEclipse wrote: post #125

I think it was always a mistake to pretend hyperthreading doubles your core count. I always assumed it was just due to laziness; the operating system treats a hyperthreaded core as two "virtual cores" and schedules as two cores, so then every other piece of tooling sees double the number of actual cores. There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages…

>There's no good reason I know of that a CPU utilization tool shouldn't use real cores when calculating percentages

On AMD, threads may as well be cores. If you take a Ryzen and disable SMT, you're basically halving its parallelism, at least for some tasks. On Intel you're just turning off an extra 10-20%.

Re: %CPU utilization is a lie

#132

Post by RaftPeople »

c2h5oh wrote: post #44
PaulKeeble wrote: post #20

This is bang on, you can't count the hyperthreads as double the performance, typically they are actually in practice only going to bring 15-30% if the job works well with it and their use will double the latency. Failing to account for loss in clockspeed as the core utilisation climbs is another way its not linear and in modern software for the desktop its really something to pay careful attention to. It should be po…

To complicate things more HT performance varies wildly between CPU architectures and workloads. e.g. AMD implementation, especially in later Zen cores, is closer to a performance of a full thread than you'd see in Intel CPUs. Provided you are not memory bandwidth starved.

> To complicate things more HT performance varies wildly between CPU architectures and workloads.

IBM's Power cpu's have also traditionally done a great job with SMT compared to Intel's implementation.

Re: %CPU utilization is a lie

#133

Post by everforward »

tracker1 wrote: post #123
ot wrote: post #7

Utilization is not a lie, it is a measurement of a well-defined quantity, but people make assumptions to extrapolate capacity models from it, and that is where reality diverges from expectations. Hyperthreading (SMT) and Turbo (clock scaling) are only a part of the variables causing non-linearity, there are a number of other resources that are shared across cores and "run out" as load increases, like memory bandwidth…

> There is rarely a stable definition of "work". Is it RPS? Request cost can vary even throughout the day. Is it instructions? Same, the typical IPC can vary. I think this is probably one of the most important points... similarly, is this public facing work dealing with any kind of user request, or is it simply crunching numbers/data to build an AI model from a stable backlog/queue? My take has always been with moder…

> In any case, for me... >= 65% CPU load for >= 30m/day means it's at 100% effective utilization, and needs expansion relatively soon.

I think this depends on workload still because IO heavy apps hyperthread well and can push up to 100%. I think most of the apps I've worked on end up being IO bound because "waiting on SQL results" or the more generic "waiting on downstream results" is 90% of their runtime. They might spend more time reading those responses off the wire than they do actually processing anything.

There are definitely things that isn't true of though, and your metrics read about right to me.

Re: %CPU utilization is a lie

#134

Post by pama »

saagarjha wrote: post #121
pama wrote: post #50

Wait until you encounter GPU utilization. You could have two codes listing 100% utilization and have well over 100x performance difference from each other. The name of these metrics creates natural assumptions that are just wrong. Luckily it is relatively easy to estimate the FLOP/s throughput for most GPU codes and then simply compare to the theoretical peak performance of the hardware.

If your workload is compute bound, of course. Sometimes you want to look at bandwidth instead.

Of course. Lots of useful metrics exist to help tweak code performance without always needing to go into detailed profiler traces. GPU utilization is a particularly poor metric in helping much, except for making sure the code made it to the GPU somehow :-)

Re: %CPU utilization is a lie

#135

Post by cpncrunch »

saagarjha wrote: post #118
kqr wrote: post #76

It might be a lie, but it surely is a practical one. In my brief foray into site reliability engineering I used CPU utilisation (of CPU-bofund tasks) with queueing theory to choose how to scale servers before big events. The %CPU suggestions ran contrary to (and were much more conservative than) the "old wisdom" that would otherwise have been used. It worked out great at much lower cost than otherwise. What I'm tryin…

40% seems quite lightly utilized tbh

I tend to use 50% as a soft target, which seems like a good compromise. Sometimes it may go a little bit over that, but if it's occasional it shouldn't be an issue.

It's not good to go much over 50% on a server (assuming half the cpus are just hyperthreads), because you're essentially relying on your load being able to share the actual cpu cores. At some point, when the load increases too much, there may not be any headroom left for sharing those physical cpus. You then get to the point where adding a little bit more load to 80% suddenly results in 95% utilization.

Re: %CPU utilization is a lie

#136

Post by Ambroisie »

hinkley wrote: post #12
judge123 wrote: post #8

This hits so close to home. I once tried to explain to a manager that a server at 60% utilization had zero room left, and they looked at me like I had two heads. I wish I had this article back then!

You also want to hit him with queueing theory. Up to a hair over 60% utilization the queuing delays on any work queue remain essentially negligible. At 70 they become noticeable, and at 80% they've doubled. And then it just turns into a shitshow from there on. The rule of thumb is 60% is zero, and 80% is the inflection point where delays go exponential. The biggest cluster I ran, we hit about 65% CPU at our target P9…

Do you have a link to a more in-depth analysis of the queuing theory for these numbers?

Re: %CPU utilization is a lie

#137

Post by tonymet »

I like his empirical approach to get to the root significance of the cpu %-age indicator. Software engineers and data analysts take discrete "data" measurements and statistics for granted.

"data" / "stats" are only a report, and that report is often incorrect.

Re: %CPU utilization is a lie

#139

Post by BrendanLong »

4gotunameagain wrote: post #99
mustache_kimono wrote: post #84

Reminds me of Brendan Gregg's "CPU Utilization is Wrong" but this blog fails to discuss that blog's key point that CPU utilization is a measure of whether or not the CPU is busy, including whether the CPU is waiting [0]. That blog also explains that the IPC (instructions per cycle) metric actually measures useful work hidden within that busy state. [0]: https://www.brendangregg.com/blog/2017-05-09/cpu-utilization...

What's up with Brendans and CPU utilisation concerns, any Brendan to shine some light ?

I'd love to explain, but you'd need to change your name to Brendan first.
Post reply on HN