The "why not use" section should probably include nvtop?
Possibly also nvitop, which is a different tool from nvtop: https://github.com/XuehaiPan/nvitop
GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
11–20 of 49 posts
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#12Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
how so?
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#13What is the benefit of this over `watch nvidia-smi`, possibly prepended with an `ssh` in the case of a remove server?
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#14This is awesome! Tested it out while running some plex encoding and everything worked as expected!
I did notice that nvidia-smi shows the process name as plex-transcoding but gpu-hot is showing [Not Found]. Not sure if that is where the process name is supposed to go
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#15The AI/vibe coded "purple" color scheme is a meme at this point - might want to tweak the look and feel to not be so on the nose, but it's otherwise a good dashboard.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#16Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
how so?
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them.
It does not at all take into count how much that thread is actually using the core to it's capacity.
So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 30 minutes for some reason, but you can see all your GPUs (except the one that cause the failure) were at 100% util, even though they clearly did nothing but wait.
Another example are operation with low compute intensity: Say you want to add 1 to every element in a very large tensor, you effectively have to transfer every element (let's say FP8, so 1 byte) from the HBM to the l2 memory, which is very slow operation, to then simply do an add, which is extremely fast. It takes about ~1000x more time to move that byte to L2 than it takes to actually do the add, so in effect your "true" utilization is ~0.2%, but nvidia-smi (and this tool) will show 100% for the entire duration of that add.
Sadly there isn't a great general way to monitor "true" utilization during training, generally you have to come up with an estimate of how many flops your model requires per pass, look at the time it takes to do said pass, and compare the flops/sec you get to Nvidia's spec sheet. If you get around 60% of theoretical flops for a typical transformer LLM training you are basically at max utilization.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#17Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a s…
CUDA toolkit comes with an occupancy calculator that can help you determine based on your kernel launch parameters how busy your GPU will potentially be.
For more information: https://docs.nvidia.com/cuda/cuda-c-programming-guide/#multi...
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#18Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
@dang sorry for the meta-comment, but why is yfontana's comment dead? I found it pretty insightful.
FYI, adding @ before a user name does nothing besides looking terrible and AFAIK dang does not get a notification when he’s mentioned. If you want to contact him, the best way is to send an email to hn@ycombinator.com .
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#19Earlier quoted context omitted.
how so?
"Utilization" tells you the percentage of your GPU's SM that currently have at least one thread assigned to them. It does not at all take into count how much that thread is actually using the core to it's capacity. So if e.g. your thread is locked waiting on some data from another GPU (NCCL) and actually doing nothing, it will still show 100% utilisation. A good way to realize that is when a NCCL call timeout after 3…
What about energy consumption as a proxy for it ?
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#20Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a s…
For kernel-level performance tuning you can use the occupancy calculator as pointed out by jplusqualt or you can profile your kernel with Nsight compute which will give you a ton of info.
But for model-wide performance, you basically have to come up with your own calculation to estimate the FLOPs required by your model and based on that figure out how well your model is maxing out the GPU capabilities (MFU/HFU).
Here is a more in-depth example on how you might do this: https://github.com/stas00/ml-engineering/tree/master/trainin...