GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
1–10 of 49 posts
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#2Hi everyone, I just built a GPU dashboard to check the utilization on NVIDIA cards directly in your browser. It also works with multiple GPUs. The idea is to have real-time metrics from a remote GPU server instead of running nvidia-smi. Let me know if you try it out!
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#3Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised.
Does not change the usefulness of this dashboard, just wanted to point it out.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#4Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
Properly measuring "GPU load" is something I've been wondering about, as an architect who's had to deploy ML/DL models but is still relatively new at it. With CPU workloads you can generally tell from %CPU, %Mem and IOs how much load your system is under. But with GPU I'm not sure how you can tell, other than by just measuring your model execution times. I find it makes it hard to get an idea whether upgrading to a stronger GPU would help and by how much. Are there established ways of doing this?
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#5The "why not use" section should probably include nvtop?
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#6This is really cool, but I tend to prefer NVtop for now.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#7Oh that’s neat. Been looking for a way to see vram temps on Linux
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#8The "why not use" section should probably include nvtop?
Fair, but I believe that this is intended for a web browser rather than a terminal.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#9Obligatory reminder that "GPU utilisation" as a percentage is meaningless metric and does not tell you how well your GPU is utilised. Does not change the usefulness of this dashboard, just wanted to point it out.
@dang sorry for the meta-comment, but why is yfontana's comment dead? I found it pretty insightful.
Re: GPU Hot: Dashboard for monitoring NVIDIA GPUs on remote servers
#10In app.py it seems like you call nvidia-smi as a subprocess and then scrape that. Are there no bindings to do that directly?