Simplifying GPU Application Development with HMM
developer.nvidia.com
Simplifying GPU Application Development with HMM
1–10 of 36 posts
Re: Simplifying GPU Application Development with HMM
#2they used to not support unified memory in their vGPU drivers. it was a major deal breaker back then.
Re: Simplifying GPU Application Development with HMM
#3> This new ability to directly read or write to the full application memory address space will significantly improve programmer productivity for all programming models built on top of CUDA: CUDA C++, Fortran, standard parallelism in Python, ISO C++, ISO Fortran, OpenACC, OpenMP, and many others.
This is the part of CUDA alternatives always miss when their models only support C and some C++ subset.
Re: Simplifying GPU Application Development with HMM
#4Does that mean that now anyone with rtx20 series or above can run local ML models as big as their RAM allows? (Or larger if they're happy to wait for swapping to SSD) Or am I misunderstanding the scale of the impact here?
(Not exactly "now", but when the software is recompiled / ported to this)
Re: Simplifying GPU Application Development with HMM
#5Does that mean that now anyone with rtx20 series or above can run local ML models as big as their RAM allows? (Or larger if they're happy to wait for swapping to SSD) Or am I misunderstanding the scale of the impact here? (Not exactly "now", but when the software is recompiled / ported to this)
You can already do that with GGUF/GGML models which allow you to split between CPU and GPU. Obviously there is a performance hit when running on your DDR5 and CPU compared to HBM/GDDR and GPU but it’s better than nothing.
Re: Simplifying GPU Application Development with HMM
#6Does that mean that now anyone with rtx20 series or above can run local ML models as big as their RAM allows? (Or larger if they're happy to wait for swapping to SSD) Or am I misunderstanding the scale of the impact here? (Not exactly "now", but when the software is recompiled / ported to this)
You could already do that with Unified Memory which has existed for a while and IIRC supported paging and swapping, assuming you `cudaMalloc` and `cudaFree` appropriately for your allocations.
This is not a change to "features" but a change to the programming model. You now never need to ever write cudaMalloc or cudaFree, you can just use any allocator or tool. This means more off the shelf code will just work when used with CUDA. So now your io_uring buffers can be shared with the GPU trivially, for example, or mmap'd pages that a library gave you, or whatever.
The programming model is one of the things Nvidia does significantly better than any competitor. Single source model + HMM is a big step up from something like OpenCL in productivity and correctness.
On Grace Hopper chips, HMM is granular down to the cache line (64 bytes); on x86 systems I believe they said it's (of course) a 4k page granularity.
Re: Simplifying GPU Application Development with HMM
#7they used to not support unified memory in their vGPU drivers. it was a major deal breaker back then.
I think that's still the case; though I assume you're talking about Linux-on-Linux vGPUs, the same is true of e.g. WSL2 where unified memory isn't supported. Sucks, because it's a great feature.
Re: Simplifying GPU Application Development with HMM
#8they used to not support unified memory in their vGPU drivers. it was a major deal breaker back then.
According to the manual, UVM is supposed to be working on vGPUs (at least MIG-backed vCS), I could never get it working though
Re: Simplifying GPU Application Development with HMM
#9>"As an aside, new hardware platforms such as NVIDIA Grace Hopper natively support the Unified Memory programming model through hardware-based memory coherence among all CPUs and GPUs. For such systems, HMM is not required, and in fact, HMM is automatically disabled there.
One way to think about this is to observe that HMM is effectively a software-based way of providing the same programming model as an NVIDIA Grace Hopper Superchip."
1) I am curious what the AMD equivalent of nVidia's HMM is, or will be...
2) I am curious if software will be able to be written with HMM (or some higher level abstraction API) such that HMM enabled software will also function on an AMD or other 3rd party GPU...
Re: Simplifying GPU Application Development with HMM
#10>"As an aside, new hardware platforms such as NVIDIA Grace Hopper natively support the Unified Memory programming model through hardware-based memory coherence among all CPUs and GPUs. For such systems, HMM is not required, and in fact, HMM is automatically disabled there. One way to think about this is to observe that HMM is effectively a software-based way of providing the same programming model as an NVIDIA Grace…
AMDs answer will be “nothing” imho.
They’ve really left this area wide open for over a decade now when it’s been extremely clear this is where the market was going.
Their GPU and GPU compute story is a mess, because rocm has the most confusing compatibility story possible . They’ve been late to compute accelerators as well.
I don’t think there’ll be any abstraction layers either. The community as a whole is more than happy to be single vendor. AMD has shown they can’t build compute stacks, not because of technology reasons but purely long term decisions. The community therefore won’t do it for them.