sudo apt install nvtop
// solves everything at the above container claims to do lol
GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
21–30 of 49 posts
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#22Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a s…
you need to profile them, nsight is one even torch does flamegraphs
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#23Earlier quoted context omitted.
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them. It does not at all take into count how much that thread is actually using the core to it's capacity. So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 3…
What about energy consumption as a proxy for it ?
not a good estimator but still roughly good, ambient temps/neighboring cards alone might influence this more than workloads
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#24Earlier quoted context omitted.
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them. It does not at all take into count how much that thread is actually using the core to it's capacity. So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 3…
What about energy consumption as a proxy for it ?
Definitely a better high level metric than nvidia-smi, and probably fine if you just want to get a very coarse idea of whether or not your are using the GPUs reasonably at all.
But when you get to the point where you care about a few percentage points of utilisation it's just not reliable enough as many things can impact energy consumption both ways. E.g. had a case were the GPU cluster we were using wasn't being cooled well enough, so you would gradually see power draw getting lower and lower as the GPUs were throttling themselves to not overheat.
You can also find cases were energy consumption is high but MFU/HFU isn't, like memory intensive workloads
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#25Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
how so?
Utilisation is counted by the OS, it's not exposed as a performance counter by the hardware. Thus, it's limited by the level of abstraction presented by the hardware.
It's useless on CPUs as well, just to a much much lesser extent to the point of it actually being useful.
Basically, the OS sees the CPU as being composed of multiple cores, that's the level of abstraction. Thus, the OS calculates "portion of last second where atleast one instruction was sent to this core" on each core and then reports it. The single number version is an average of each core's value.
On the other hand, the OS cannot calculate stuff inside each core - the CPU hides as part of its abstraction. That is, you cannot know "I$ utilisation", "FPU utilisation", etc,.
In the GPU, the OS doesn't even see each SM (streaming multiprocessor, loosely analogous to a cpu core). It just sees the whole GPU as one black box abstraction. Thus, it calculates utilisation as "portion of last second where atleast one kernel was executing on the whole GPU". It cannot calculate intra-GPU util at all. So one kernel executing on one SM looks the same to the OS, as that kernel executing on tens of SMs!
This is the crux of the issue.
With performance counters (perf for CPU, or nsight compute for GPU), lots of stuff visible only inside the hardware abstraction can be calculated (SM util, warp occupancy, tensor util, etc)
The question then, is why doesn't the GPU schedule stuff on each SM in the OS/driver? Instead of doing it in a microcontroller in the hardware itself on the other side of the interface?
Well, I think it's due to efficiency reasons and also for nvidia to have more freedom to change it without having compat issues due to being tied to the OS, and similar reasons. If that were the case however, then the OS could calculate util for each SM, and then average it, giving you more accurate values - the case with the kernel running on 1 SM will report a smaller util than the case with the kernel executing on 15 SMs.
IME, measuring on nsight compute causes anywhere from a 5% to 30% performance overhead, so if that's ok for you, you can enable it and get more useful measurements.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#26Earlier quoted context omitted.
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them. It does not at all take into count how much that thread is actually using the core to it's capacity. So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 3…
What about energy consumption as a proxy for it ?
iirc most of the energy comes from memory IO not arithmetic, so it's still not great. A better direction, though.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#27another option is to use Prometheus+grafana https://docs.nvidia.com/datacenter/cloud-native/gpu-telemetr...
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#28The "why not use" section should probably include nvtop?
Absolutely.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#29Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a s…
It's harder than measuring CPU load, and depends a lot on context. For example, often 90% of a GPU's available flops are exclusively for low-precision matrix multiply-add operations. If you're doing full precision multiply-add operations at full speed, do you count that as 10% or 100% load? If you're doing lots of small operations and your warps are only 50% full, do you count that as 50% or 100% load? Unfortunately, there isn't really a shortcut to understanding how a GPU works and knowing how you're using it.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#30The AI/vibe coded "purple" color scheme is a meme at this point - might want to tweak the look and feel to not be so on the nose, but it's otherwise a good dashboard.
nah I like it