GPU utilization can be a misleading metric
1–10 of 37 posts
Re: GPU utilization can be a misleading metric
#2running GPU models and maximizing utilization is pretty opaque to me as a layman coming into the scene.
take this example: https://gist.github.com/sergiotapia/efc9b3f7163ba803a260b481... - running a fairly simple model that takes only 70ms per image pair, but because I have 300 images it becomes a big time sink.
by using ThreadPoolExecutor, I cut that down to about 16 seconds. i wonder if there is a fairly obvious way to truly utlize my beefy L40S GPU! is it MPS? I haven't been successful at even running the MPS daemon on my linux server yet. very opaque for sure!
Re: GPU utilization can be a misleading metric
#3running GPU models and maximizing utilization is pretty opaque to me as a layman coming into the scene. take this example: https://gist.github.com/sergiotapia/efc9b3f7163ba803a260b481... - running a fairly simple model that takes only 70ms per image pair, but because I have 300 images it becomes a big time sink. by using ThreadPoolExecutor, I cut that down to about 16 seconds. i wonder if there is a fairly obvious wa…
Batch as many requests together as possible and your utilization will increase.
Re: GPU utilization can be a misleading metric
#4running GPU models and maximizing utilization is pretty opaque to me as a layman coming into the scene. take this example: https://gist.github.com/sergiotapia/efc9b3f7163ba803a260b481... - running a fairly simple model that takes only 70ms per image pair, but because I have 300 images it becomes a big time sink. by using ThreadPoolExecutor, I cut that down to about 16 seconds. i wonder if there is a fairly obvious wa…
totally agreed. A lot of our findings during this process is that there's still a lot of alpha in finding the right kernels for the job/model. We're hoping that in the future `torch.compile` will become more mature because current docs on performance at least on pytorch side definitely leave us wanting more
Re: GPU utilization can be a misleading metric
#5> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations
Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this.
On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.
Re: GPU utilization can be a misleading metric
#6> you can get 100% GPU utilization by just reading/writing to memory while doing 0 computations Indeed! Utilization is a proxy for what you actually want (which is good use of available hardware). 100% GPU utilization doesn't actually indicate this. On the other hand, if you aren't getting 100% GPU utilization, you aren't making good use of the hardware.
Yup, similar to SM efficiency in that sense too. If you aren't seeing >80%, there is certainly time left on the table. But getting a high SM efficiency value doesn't guarantee you're making good use of the hardware as well. (still a better proxy than GPU util though)
Re: GPU utilization can be a misleading metric
#7Application-specific metrics are the way to go. For ML training this is one example:
https://cloud.google.com/blog/products/ai-machine-learning/g...
Re: GPU utilization can be a misleading metric
#8When understanding the performance of your model it's very helpful to look at a roofline plot [1]. The roofline plot will show you the floating-point performance as a function of arithmetic intensity for the various ops in your model. The plot has two regimes: a memory-bound regime on the left and a compute-bound regime on the right. This can help to identify memory-bound ops that are taking a significant fraction of compute time.
Re: GPU utilization can be a misleading metric
#9When understanding the performance of your model it's very helpful to look at a roofline plot [1]. The roofline plot will show you the floating-point performance as a function of arithmetic intensity for the various ops in your model. The plot has two regimes: a memory-bound regime on the left and a compute-bound regime on the right. This can help to identify memory-bound ops that are taking a significant fraction of…
Agreed, roofline plots would be quite powerful in this context. From a quick search, seems like the only way to create a roofline plot for your model would be to use Nsight [1]? Would be interested to know if there are any simpler tools, since one of the big benefits of SM efficiency is how easily the metric is accessed.
[1]: https://www.nvidia.com/en-us/on-demand/session/gtcspring21-s...
Re: GPU utilization can be a misleading metric
#10Application-specific metrics are the way to go. For ML training this is one example: https://cloud.google.com/blog/products/ai-machine-learning/g...
Nice, seems like ML Productivity Goodput is a pretty well thought-out metric to understand the overall efficiency of your cluster. I'll consider adding this into our cluster management platform. Only potential drawbacks I'd guess are it being somewhat difficult to compute since it relies on metrics like MFUs, and not something we can observe layer-by-layer to understand inefficient kernels, but I'll take a deeper look. Thanks!