As someone that is familiar with using nvidia-smi to track util, what are some commands people use to track the SM efficiency? The end of the article had some references, but no examples of what to use explicitly.
GPU utilization can be a misleading metric
31–37 of 37 posts
Re: GPU utilization can be a misleading metric
#32I’ve recently been trusting gpu watt usage over utilization. Any idea how good that is as a simple proxy (if I’m just looking at nvidia-smi)?
Power usage is indeed a better representation of GPU utilization during ML training. It has the advantage of combining many important indirect signals that aren’t visible, and avoids many downfalls of compute usage, which can go to 100% even in all-reduce deadlocks, among other scenarios.
Re: GPU utilization can be a misleading metric
#33I’ve recently been trusting gpu watt usage over utilization. Any idea how good that is as a simple proxy (if I’m just looking at nvidia-smi)?
power is also a good proxy. For example, we've had distributed runs that we monitored on WandB where one of our workers died in the middle and the rest were basically stalling on the dead worker. On WandB, we were only logging GPU stats on one worker and that one had 100% util but basically no excess power draw compared to having nothing running, which is how I found out something was stalling. Restarting fixed it and got the power draw up to normal, but even with high power draw, we were still having some sections of code with low SM efficiency (~20%) for that training.
Re: GPU utilization can be a misleading metric
#34"If we have a CUDA kernel that continuously runs for 10 seconds but only uses 1 SM, on an H100, this would register 100% utilization, but the SM efficiency would be 1 / 132 = 0.7%." does this situation register 100% utilization? BTW, the SM OCCUPANCY is also a metric you need to care about if you concern on kernel efficiency
That's why I look mostly at the H100 temperatures. Gives a better utilization metric
Re: GPU utilization can be a misleading metric
#35As someone that is familiar with using nvidia-smi to track util, what are some commands people use to track the SM efficiency? The end of the article had some references, but no examples of what to use explicitly.
Unfortunately, SM efficiency is not accessible via nvidia-smi. The best methods to track it would be to:
1. Profile your model with Pytorch Profiler 2. Export metrics with Nvidia DCGM
Re: GPU utilization can be a misleading metric
#36Earlier quoted context omitted.
Not so inherently IMO. What I mean is: where did you take that from? I program FFTs on GPUs, and I see no reason for the "inherently can't reach 100% utilization by any metric".
I interpret that comment as you're not going to be using every silicon block that the GPU provides, like video codecs and rasterizing. If you've maxed out compute without going over the power budget, for example, you'd likely still be able to decode video if the GPU has a separate block for it.
I had a similar read .. I packed a lot of parallel FFT's and other processing into custom TI DSP cards but the DSP family chips were RISC and carried little 'baggage' - just fat fat 32 bit | 64 bit floating point pipelines with instruction sets optimised for modular ring indexing of scalar | vector operations.
Even then they ran @ 80% "by design" for expected hard real time usage .. they only went to 11 and dropped results in toast until they smoke tests and with operators that redlined limits (and got feedback to that effect).
Re: GPU utilization can be a misleading metric
#37> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this. On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.
This sounds like the good old "having high test coverage is bad because I can get to 100% just by calling functions and doing nothing, asserting nothing with them". 100% test coverage doesn't mean your tests are good, but having 50% (or pick your number) means they are bad / not sufficient.
That isn't even necessarily true. For interpreted languages having a test that just runs code asserts that the code is able to run (i.e. you are not calling a string object as a function for example). Which is not enough to always assert functionality but still better than nothing.