Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

%CPU utilization is a lie

brendanlong.com

Post reply on HN
Print view View original

161–170 of 176 posts

Re: %CPU utilization is a lie

#161

Post by swiftcoder »

kristianp wrote: post #160
swiftcoder wrote: post #89

I remember being stuck in a discussion with management one time, that went something like this: Manager: CPU utilisation is 100% under load! We have to migrate to bigger instances. Me: but is the CPU actually doing useful work? (chat, it was not. busy waiting is CPU utilisation too)

How do you measure the amount of busy waiting?

I don't think there is a good general tool for this. In this specific case, I went spelunking for all the points where we had thread contention over resources, and discovered that for several resources quite a lot of CPU cycles were being expended to no use. The goal is really to eliminate the underlying resource contention - we added per-thread caches I various places, swapped out the logging system, and were able to ~double the system throughput during times when top showed the system to be "fully loaded"

Re: %CPU utilization is a lie

#162

Post by 1718627440 »

SAI_Peregrinus wrote: post #143
ot wrote: post #129

Earlier quoted context omitted.

It is a linear percentage of the amount of time the CPU is not idle. It is not linear in the amount of useful work, but that's not what "utilization" means. The lie is the assumption that CPU time is linear in useful work, but that has nothing to do with the definition of utilization, it's just something that people sometimes naively believe. > CPU utilization isn't a lie, % CPU utilization is What do you mean by thi…

Utilization can never reach 100%, since not all of the components of the CPU (or even one core) can actually be in use at once. Quite a few are shared between operations and thus mutually exclusive.

It's of course time. It's 1-X all components of the CPU were idle.

Re: %CPU utilization is a lie

#163

Post by magicalhippo »

porridgeraisin wrote: post #159
magicalhippo wrote: post #158

Earlier quoted context omitted.

I wanted the full power when doing long compiles and such. Just not the fan jojo acion when neowsing the web or writing. Also swapping the governor was trivial and reliable. Modifying fan profiles has always been a bit of a struggle for me, with huge differences in hardware support, persistence etc.

> jojo acion when neowsing jumping into action when browsing

Thanks. I hate the autocorrect as it's so often wrong, but hitting the right "keys" with no tactile feedback is such a pain. I miss T9...

Re: %CPU utilization is a lie

#164

Post by aaa_2006 »

CPU utilization alone is misleading. Pair it with per core load average or runqueue length to see how threads are actually queuing. That view often reveals the real bottleneck, whether it is I/O, memory, or scheduling delays.

Re: %CPU utilization is a lie

#165

Post by BrendanLong »

BrendanLong wrote: post #73
BrendanLong wrote: post #70

Earlier quoted context omitted.

The advantage of stress-ng is that it's easy to make it run with specific CPU utilization numbers. The tests where I run some number of workers at 100% utilization are interesting since they give such perfect graphs, but I think the version where I have 24 workers and increase their utilization slowly is more realistic for showing how production CPU utilization changes.

Fun data point though, I just ran three data points of the Phoronix nginx benchmark and got these results: - Pinned to 6 cores: 28k QPS - Pinned to 12 cores: 56k QPS - All 24 cores: 62k QPS I'm not sure how this applies to realistic workloads where you're using all of the cores but not maxing them out, but it looks like hyperthreading only adds ~10% performance in this case.

Here's results of the Nginx benchmark pinned to 1-24 cores: https://docs.google.com/spreadsheets/d/1d_OK_ckLT1zTA_fG4vkq...

At 51% reported CPU utilization, it's doing about 80% of the maximum requests per second, and it can't get above 80% utilization.

I also added a section: https://www.brendanlong.com/cpu-utilization-is-a-lie.html#bo...

Re: %CPU utilization is a lie

#166

Post by nomel »

kiitos wrote: post #153
kristopolous wrote: post #141

Earlier quoted context omitted.

This was the same interview where some guy was asking me about "big-o" - like the thing that you teach 19 year olds and I was saying that parallelization matters, i/o matters, quantization matters, whether you can run it on the GPU, these all matter. The simple "big-o" number doesn't account for whether you need to pass terabytes over the bus for every operation - and on actual computers moving around terabytes, I kn…

"big o" usually refers to algorithmic complexity, which is something entirely orthogonal to all of the dimensions you mentioned obviously all of this stuff matters in the end but big-o comes before all of those other things

> but big-o comes before all of those other things

If you're attempting to quantify algorithmic scalability with big-o, without those in mind, you'll often be wrong. There was a great post here a few years ago going into this, and how memory access "complexity" is what usually matters, and what dominantly shapes the scalability curve. It had nice examples showing how the expected big-o scalability curves were often completely wrong, outside of toys.

If you're not trying to quantify algorithmic scalability with big-o, then have fun coming up with a fun collection of symbols to put next to your code, and petting your spherical cow!

Re: %CPU utilization is a lie

#167

Post by kiitos »

nomel wrote: post #166
kiitos wrote: post #153

Earlier quoted context omitted.

"big o" usually refers to algorithmic complexity, which is something entirely orthogonal to all of the dimensions you mentioned obviously all of this stuff matters in the end but big-o comes before all of those other things

> but big-o comes before all of those other things If you're attempting to quantify algorithmic scalability with big-o, without those in mind, you'll often be wrong. There was a great post here a few years ago going into this, and how memory access "complexity" is what usually matters, and what dominantly shapes the scalability curve. It had nice examples showing how the expected big-o scalability curves were often c…

algorithmic complexity is 100% absolutely orthogonal to the stuff you've mentioned

what you're describing is something different than big-o, in the sense that is commonly understood, and what your interviewer almost certainly intended

I understand what you're describing and talking about but it's not big-o

I would guess that you haven't had any kind of formal cs education? no shade but like there are some important topics covered in those curriculums

Re: %CPU utilization is a lie

#168

Post by perching_aix »

ot wrote: post #129
SAI_Peregrinus wrote: post #127

Earlier quoted context omitted.

That's all true, and the % part is still a lie. As you note, CPU utilization isn't linear, and percentages are linear measures. CPU utilization isn't a lie, % CPU utilization is.

It is a linear percentage of the amount of time the CPU is not idle. It is not linear in the amount of useful work, but that's not what "utilization" means. The lie is the assumption that CPU time is linear in useful work, but that has nothing to do with the definition of utilization, it's just something that people sometimes naively believe. > CPU utilization isn't a lie, % CPU utilization is What do you mean by thi…

Admittedly, I'm not there on the industry frontlines reading (or writing) whitepapers on CPU design, so my knowledge on CPU internals is fairly limited. Here's the premise I'm working with:

- operations are implemented in different sub-units of each core

- operations are pipelined, to help saturate these sub-units, so multiple ops executing on different sub-units can be in-flight at the same time

- operations are reordered and their execution is predicted, to help saturate the pipelines

Given all of these, to report the overall saturation of each core sounds like quite the challenge. It'd mean collecting data on how busy each sub-unit is, versus how busy it could be, then weighing that against how saturated the pipelines are leading there. Maybe one sub-unit is being fed to its brink, but another could still be fed work, it just isn't: maybe the program cannot do so, or isn't willing to do so, doesn't matter.

And so none of this would show up on the scheduler I believe. From the scheduler data, you get the assignment saturation, and then whatever the CPU ended up executing is whatever it did. Did it only do integer math? Did it only do matrix math? Busy spin? Something else? Maybe most sub-units remained completely dark. It's not a utilization ratio then though, but an assignment ratio. How much time each logical core spent assigned work, versus how much it didn't.

Provided I'm not off-base, I really don't find this to be a matter of "naivity" then on people's part. It's an honestly incorrect use of language. Regardless of the reason, e.g. if the kernel cannot actually determine the kind of utilization I describe, or if it doesn't make sense on a fundamental level somehow to try to, this still doesn't justify torturing the language by calling this utilization. It could be just referred to what it is then: assignment. This is like the difference between reserved and committed memory. Or like the difference between me working from 9-5, and me being on meetings from 9-1 and working from 1-5.

Re: %CPU utilization is a lie

#169

Post by nomel »

kiitos wrote: post #167
nomel wrote: post #166

Earlier quoted context omitted.

> but big-o comes before all of those other things If you're attempting to quantify algorithmic scalability with big-o, without those in mind, you'll often be wrong. There was a great post here a few years ago going into this, and how memory access "complexity" is what usually matters, and what dominantly shapes the scalability curve. It had nice examples showing how the expected big-o scalability curves were often c…

algorithmic complexity is 100% absolutely orthogonal to the stuff you've mentioned what you're describing is something different than big-o, in the sense that is commonly understood, and what your interviewer almost certainly intended I understand what you're describing and talking about but it's not big-o I would guess that you haven't had any kind of formal cs education? no shade but like there are some important t…

I have. I understand big-o, I understand that it’s just algorithmic complexity. I understand big-o is not a performance scaling model, because algorithms run on real hardware. That's fine. Some people enjoy petting spherical cows, and some people work with the nuances of reality. That's also fine.

Re: %CPU utilization is a lie

#170

Post by p12tic »

tracker1 wrote: post #123
ot wrote: post #7

Utilization is not a lie, it is a measurement of a well-defined quantity, but people make assumptions to extrapolate capacity models from it, and that is where reality diverges from expectations. Hyperthreading (SMT) and Turbo (clock scaling) are only a part of the variables causing non-linearity, there are a number of other resources that are shared across cores and "run out" as load increases, like memory bandwidth…

> There is rarely a stable definition of "work". Is it RPS? Request cost can vary even throughout the day. Is it instructions? Same, the typical IPC can vary. I think this is probably one of the most important points... similarly, is this public facing work dealing with any kind of user request, or is it simply crunching numbers/data to build an AI model from a stable backlog/queue? My take has always been with moder…

> Today it's a bit more complicated when you have servers with 100+ cores as an option for under $30k (guestimate based on $10k CPU price).

If one can buy used, then previous generation 128C 256T epyc server is less than $5k. For homelabs that can accept non-rackmount gear it's less than $3k.

Post reply on HN