GPU utilization can be a misleading metric • Hacker News Board

Re: GPU utilization can be a misleading metric

#21

Post by tanelpoder » Fri, Aug 23, 2024, 5:27 AM UTC

SnowflakeOnIce wrote: post #5

> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this. On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.

This reminds me of the Linux/Unix disk busy "%util" metric in tools like sar and iostat. People sometimes interpret the 100%util as a physical ceiling for the disk IO capacity, just like with CPUs ("we need more disks to get disk I/O utilization down!").

It is a correct metric when your block device has a single physical spinning disk that can only accept one request at a time (dispatch queue depth=1). But the moment you deal with SSDs (capable of highly concurrent NAND IO), SAN storage block devices striped over many physical disks or even a single spinning disk that can internally queue and reorder IOs for more efficient seeking, just hitting 100%util at the host block device level doesn't mean that you've hit some IOPS ceiling.

So, looks like the GPU "SM efficiency" analysis is somewhat like logging in to the storage array itself and checking how busy each physical disk (or at least each disk controller) inside that storage array is.

Re: GPU utilization can be a misleading metric

#22

Post by AeZ1E » Fri, Aug 23, 2024, 8:01 AM UTC

gpu utilization is not everything, people! mfus are where it's at. time to recalibrate those expectations and tap into the true potential of your gpus. brace yourselves, the real efficiency is yet to come!

Re: GPU utilization can be a misleading metric

#23

Post by jorvi » Fri, Aug 23, 2024, 10:37 AM UTC

SnowflakeOnIce wrote: post #5

> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this. On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.

> On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.

Some of us like having more than 2 hours of battery life, and not scalding our skin in the process of using our devices.

Re: GPU utilization can be a misleading metric

#24

Post by jfkfif » Fri, Aug 23, 2024, 11:18 AM UTC

roanakb wrote: post #9

antognini wrote: post #8

When understanding the performance of your model it's very helpful to look at a roofline plot [1]. The roofline plot will show you the floating-point performance as a function of arithmetic intensity for the various ops in your model. The plot has two regimes: a memory-bound regime on the left and a compute-bound regime on the right. This can help to identify memory-bound ops that are taking a significant fraction of…

Agreed, roofline plots would be quite powerful in this context. From a quick search, seems like the only way to create a roofline plot for your model would be to use Nsight [1]? Would be interested to know if there are any simpler tools, since one of the big benefits of SM efficiency is how easily the metric is accessed. [1]: https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s...

Depending on the size of your application you can calculate flops by hand

https://docs.nersc.gov/tools/performance/roofline/

Re: GPU utilization can be a misleading metric

#25

Post by mpreda » Fri, Aug 23, 2024, 11:19 AM UTC

shaklee3 wrote: post #20

defrost wrote: post #19

Earlier quoted context omitted.

In remote sensing | computation physicas applications it's rare to have a single FFT to compute (whatever algorithm is chosen). Hence the practice of stuffing many FFT's through GPU grids in parallel and working to max out the hardware usage in order to increase application throughput. eg: https://arxiv.org/pdf/1707.07263 https://ieeexplore.ieee.org/document/9835388

I don't mean a single fft. I mean the fft algorithms are inherently not going to use the GPU at 100% utilization by any metric.

Not so inherently IMO.

What I mean is: where did you take that from? I program FFTs on GPUs, and I see no reason for the "inherently can't reach 100% utilization by any metric".

Re: GPU utilization can be a misleading metric

#26

Post by danielvaughn » Fri, Aug 23, 2024, 12:02 PM UTC

We ran into a similar problem with CPU utilization at my job. Created an alert for when our systems hit 90% CPU util, and ended up with a ton of noise. We realized that for some of our workloads, this was normal and expected.

Re: GPU utilization can be a misleading metric

#27

Post by lights0123 » Fri, Aug 23, 2024, 12:15 PM UTC

mpreda wrote: post #25

shaklee3 wrote: post #20

Earlier quoted context omitted.

I don't mean a single fft. I mean the fft algorithms are inherently not going to use the GPU at 100% utilization by any metric.

Not so inherently IMO. What I mean is: where did you take that from? I program FFTs on GPUs, and I see no reason for the "inherently can't reach 100% utilization by any metric".

I interpret that comment as you're not going to be using every silicon block that the GPU provides, like video codecs and rasterizing. If you've maxed out compute without going over the power budget, for example, you'd likely still be able to decode video if the GPU has a separate block for it.

Re: GPU utilization can be a misleading metric

#28

Post by shaklee3 » Fri, Aug 23, 2024, 1:06 PM UTC

mpreda wrote: post #25

shaklee3 wrote: post #20

Earlier quoted context omitted.

I don't mean a single fft. I mean the fft algorithms are inherently not going to use the GPU at 100% utilization by any metric.

Not so inherently IMO. What I mean is: where did you take that from? I program FFTs on GPUs, and I see no reason for the "inherently can't reach 100% utilization by any metric".

I'd be curious to see how you can do it. Try launching an fft of any size and batches and see if you can hit 100%

Re: GPU utilization can be a misleading metric

#29

Post by areichenbach » Fri, Aug 23, 2024, 1:16 PM UTC

I’ve recently been trusting gpu watt usage over utilization. Any idea how good that is as a simple proxy (if I’m just looking at nvidia-smi)?

Re: GPU utilization can be a misleading metric

#30

Post by HPsquared » Fri, Aug 23, 2024, 1:30 PM UTC

serial_dev wrote: post #13

SnowflakeOnIce wrote: post #5

> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this. On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.

This sounds like the good old "having high test coverage is bad because I can get to 100% just by calling functions and doing nothing, asserting nothing with them". 100% test coverage doesn't mean your tests are good, but having 50% (or pick your number) means they are bad / not sufficient.

In other words it's "necessary, but not sufficient".