Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simplifying GPU Application Development with HMM

developer.nvidia.com

Post reply on HN
Print view View original

21–30 of 36 posts

Re: Simplifying GPU Application Development with HMM

#21

Post by pjmlp »

mratsim wrote: post #16
pjmlp wrote: post #3

> This new ability to directly read or write to the full application memory address space will significantly improve programmer productivity for all programming models built on top of CUDA: CUDA C++, Fortran, standard parallelism in Python, ISO C++, ISO Fortran, OpenACC, OpenMP, and many others. This is the part of CUDA alternatives always miss when their models only support C and some C++ subset.

Apple Metal does this though.

You missed the polyglot description regarding which workloads CUDA supports.

Re: Simplifying GPU Application Development with HMM

#22

Post by drewg123 »

I used to work on Myrinet HPC NICs many years ago, and the ability for a PCI(e) device to access any memory by user virtual address was a desirable feature. I believe that Quadrics did this first using a patched version of DEC OSF/1 (UNIX, Tru64, whatever you want to call it), where they hooked into the kernel pmap (page table) code, and sync'ed the page tables with their NIC. That way the NIC could do the virtual to physical translations, and know if a virtual memory address was backed by a physical page.

What Nvidia is doing here sounds similar. Does linux provide such primitives now?

Re: Simplifying GPU Application Development with HMM

#23

Post by dragontamer »

amelius wrote: post #20

"What every programmer should know about memory" needs an update. https://people.freebsd.org/~lstewart/articles/cpumemory.pdf

I don't think so.

The only thing that's been added is bank-groups in DDR4 IMO. But all you need to know is that modern RAM is maybe 16x to 32x way parallel per stick. The interface operates are faster than RAM can respond in time, so an "Optimal" CPU will list off 32x to 64x (32x for the first stick, 32x for the 2nd stick) read/write commands before the first command ever responds.

Understanding that mechanism is what that document is about (how CPUs coalesce memory and parallelizes requests).

----------------

GPUs have one additional coalesce layer given channel vs bank conflicts, and all that noise. But most GPU manuals (be they NVidia or AMD) will cover those details.

Re: Simplifying GPU Application Development with HMM

#24

Post by drewg123 »

drewg123 wrote: post #22

I used to work on Myrinet HPC NICs many years ago, and the ability for a PCI(e) device to access any memory by user virtual address was a desirable feature. I believe that Quadrics did this first using a patched version of DEC OSF/1 (UNIX, Tru64, whatever you want to call it), where they hooked into the kernel pmap (page table) code, and sync'ed the page tables with their NIC. That way the NIC could do the virtual to…

Its really hard to google for information on older stuff like this. I did find a presentation from 2000 where they talk about "OS Bypass with Virtual Addressing; no page locking or copying; full protection" (https://hsi.web.cern.ch/HNF-Europe/sem3_2001/hnf.pdf)

Re: Simplifying GPU Application Development with HMM

#25

Post by Me1000 »

capableweb wrote: post #18
mratsim wrote: post #16

Earlier quoted context omitted.

Apple Metal does this though.

Although Apple holds a <10% of total market share when it comes to computers, so not sure how helpful it is.

Well Nvidia holds like 90% of the GPU marketshare, so any reply mentioning a competitor would have this property.

Re: Simplifying GPU Application Development with HMM

#26

Post by Havoc »

viraptor wrote: post #4

Does that mean that now anyone with rtx20 series or above can run local ML models as big as their RAM allows? (Or larger if they're happy to wait for swapping to SSD) Or am I misunderstanding the scale of the impact here? (Not exactly "now", but when the software is recompiled / ported to this)

Already doable. The gotcha is that it is slow AF. Even if it’s 90%/10% split the subjective experience tanks hard so usually makes sense to pick something that fits into your vram

Re: Simplifying GPU Application Development with HMM

#27

Post by 0cf8612b2e1e »

smcleod wrote: post #5
viraptor wrote: post #4

Does that mean that now anyone with rtx20 series or above can run local ML models as big as their RAM allows? (Or larger if they're happy to wait for swapping to SSD) Or am I misunderstanding the scale of the impact here? (Not exactly "now", but when the software is recompiled / ported to this)

You can already do that with GGUF/GGML models which allow you to split between CPU and GPU. Obviously there is a performance hit when running on your DDR5 and CPU compared to HBM/GDDR and GPU but it’s better than nothing.

I have not been keeping up with developments. Does this mean mortals can run the biggest tier of Llama models (albeit with trash performance) by using system ram? For playing around, I would be willing to let my system chug along just to see what the top tier models can achieve.

Re: Simplifying GPU Application Development with HMM

#28

Post by JonChesterfield »

peter_d_sherman wrote: post #9

>"As an aside, new hardware platforms such as NVIDIA Grace Hopper natively support the Unified Memory programming model through hardware-based memory coherence among all CPUs and GPUs. For such systems, HMM is not required, and in fact, HMM is automatically disabled there. One way to think about this is to observe that HMM is effectively a software-based way of providing the same programming model as an NVIDIA Grace…

HMM is a Linux thing, not an nVidia thing. https://www.kernel.org/doc/html/v5.0/vm/hmm.html

AMD has much the same variations as nvidia here, some details at https://github.com/amd/amd-lab-notes/blob/release/mi200-memo.... The single memory systems are called APUs. The internet thinks the MI300 (in El Capitan) is one of those. The games consoles and mobile chips are too.

I'm not sure what the limits are in terms of arbitrary heterogenous execution if you want to push the boundaries, e.g. can you JIT amdgpu code into memory you got from mmap and have one of the GPU execution units branch to it? I don't see why not, but haven't tried it.

In principle I suppose a page should be able to migrate between nvidia and amdgpu hardware on a machine containing GPUS from both vendors, though that isn't likely to be a well tested path.

Re: Simplifying GPU Application Development with HMM

#29

Post by jdoerfert »

aseipp wrote: post #15
chesp3 wrote: post #11

Earlier quoted context omitted.

HMM is, I believe, a Linux feature. AMD added HMM support in ROCm 5.0 according to this: https://github.com/RadeonOpenCompute/ROCm/blob/develop/CHANG...

Note: that isn't the same thing as what the OP describes, at least according to those release notes, but it does fall under the "HMM" umbrella. You still need to specifically allocate your memory with hipMallocManaged before it can be transparently used between the CPU and GPU. Nvidia calls this "unified memory" (and has had it for 10 years now.) It's confusing, because there are basically three levels of "Heterogene…

This is not true.

3. is supported by AMD on new hardware, e.g., Frontier. See https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#...

Re: Simplifying GPU Application Development with HMM

#30

Post by trws »

aseipp wrote: post #15
chesp3 wrote: post #11

Earlier quoted context omitted.

HMM is, I believe, a Linux feature. AMD added HMM support in ROCm 5.0 according to this: https://github.com/RadeonOpenCompute/ROCm/blob/develop/CHANG...

Note: that isn't the same thing as what the OP describes, at least according to those release notes, but it does fall under the "HMM" umbrella. You still need to specifically allocate your memory with hipMallocManaged before it can be transparently used between the CPU and GPU. Nvidia calls this "unified memory" (and has had it for 10 years now.) It's confusing, because there are basically three levels of "Heterogene…

I can see how you got here from the release notes, but the conclusions are a bit off. For hardware and kernels that support the full HMM setup with AMD, you get 3 today as long as XNACK is turned on. Systems like Frontier have been using it for some time now.

Also, 2 can be subdivided into systems that implement it by having two allocations, one host one device, and triggering transfers when the GPU might access memory (2.1) and those that implement demand paging (2.2). The HMM support adds demand paging for type 2.2 as well as type 3 on supported hardware, where without it hip had to use either 2.1 or remote PCIE access to provide “unified memory”. Those were dark days, but for current hardware on appropriate kernels appropriately configured, AMD implements memory just as unified as either NVIDIA’s HMM or ATS implementations.

Post reply on HN