Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simplifying GPU Application Development with HMM

developer.nvidia.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Simplifying GPU Application Development with HMM

#31

Post by aseipp »

jdoerfert wrote: post #29
aseipp wrote: post #15

Earlier quoted context omitted.

Note: that isn't the same thing as what the OP describes, at least according to those release notes, but it does fall under the "HMM" umbrella. You still need to specifically allocate your memory with hipMallocManaged before it can be transparently used between the CPU and GPU. Nvidia calls this "unified memory" (and has had it for 10 years now.) It's confusing, because there are basically three levels of "Heterogene…

This is not true. 3. is supported by AMD on new hardware, e.g., Frontier. See https://docs.olcf.ornl.gov/systems/frontier_user_guide.html#...

Amazing, thanks for the correction(s)!

Re: Simplifying GPU Application Development with HMM

#32

Post by jhj »

For performance, it's always better to explicitly manage GPU memory and host/device copies for performance than to depend upon the unified memory paging mechanism, if it's possible to go the extra effort.

My feeling is that unified memory and on-demand paging introduced with Pascal? was mainly about making it easier to onboard existing applications (e.g., HPC codes etc) to the GPU a bit at a time with less problem. For writing a GPU application from scratch, I don't think it makes much sense (unless the granluarity of the data that you are moving around is really tiny and/or you can't predict what you would need in advance on CPU or GPU).

Re: Simplifying GPU Application Development with HMM

#33

Post by smcleod »

0cf8612b2e1e wrote: post #27
smcleod wrote: post #5

Earlier quoted context omitted.

You can already do that with GGUF/GGML models which allow you to split between CPU and GPU. Obviously there is a performance hit when running on your DDR5 and CPU compared to HBM/GDDR and GPU but it’s better than nothing.

I have not been keeping up with developments. Does this mean mortals can run the biggest tier of Llama models (albeit with trash performance) by using system ram? For playing around, I would be willing to let my system chug along just to see what the top tier models can achieve.

Technically yes - if you have lots of ram you can use that and your CPU, as you say, the performance would be pretty poor, though, especially as it’s a toll where you want to tweak your responses quite frequently. I’ve been running and old Nvidia Tesla P100 card. I got cheap on eBay for awhile now it has 16 GB of VRAM but it is pretty old. I’m so interested in this now I’ve gone out and got myself a secondhand RTX 3090 - something I never thought I’d do, but I’d really like to run 30B models in GPU.

Re: Simplifying GPU Application Development with HMM

#35

Post by _w1tm »

0cf8612b2e1e wrote: post #27
smcleod wrote: post #5

Earlier quoted context omitted.

You can already do that with GGUF/GGML models which allow you to split between CPU and GPU. Obviously there is a performance hit when running on your DDR5 and CPU compared to HBM/GDDR and GPU but it’s better than nothing.

I have not been keeping up with developments. Does this mean mortals can run the biggest tier of Llama models (albeit with trash performance) by using system ram? For playing around, I would be willing to let my system chug along just to see what the top tier models can achieve.

Yes. I recently benchmarked the 70B Llama 2 model on a 24 vCPU vSphere host with 64GB RAM (through Ollama) and it was capable of spitting out ~0.15 tokens / second. Useless for any interactive use-case but better than nothing. As a comparison the 7B Llama 2 model was ~1.5 tokens / second on the same hardware while the cheapest M1 MacBook Air can do ~10 tokens / second thanks to GPU acceleration.

Re: Simplifying GPU Application Development with HMM

#36

Post by dagmx »

kimixa wrote: post #13
dagmx wrote: post #10

Earlier quoted context omitted.

AMDs answer will be “nothing” imho. They’ve really left this area wide open for over a decade now when it’s been extremely clear this is where the market was going. Their GPU and GPU compute story is a mess, because rocm has the most confusing compatibility story possible . They’ve been late to compute accelerators as well. I don’t think there’ll be any abstraction layers either. The community as a whole is more than…

ROCm already supports HMM. You're not helping anything by going off on some rant based on an assumption and falsehood - this sort of comment is exactly the sort of thing the phrase "FUD" is used to describe.

You’re right that my rant is incorrect on the premise that they don’t have hmm, but it’s because I missed rocm adding it two years ago. So my bad, and unfortunately I can’t edit my post so I’ll leave the link here with my apologies. https://www.phoronix.com/news/Radeon-ROCm-4.3

The reason I missed it is because rocm dropped support for my cards very unceremoniously. At which point I gave up.

I do think the rest of my point outside of the first sentence is valid though. Rocm isn’t reliable to target. Nowhere near CUDA.

That it’s so dependent on what card you have, what OS/kernel you use and is so aggressive with dropping support for older cards, makes the entire ecosystem a mess. CUDA by comparison is so much more ubiquitous.

That becomes chicken and egg with popular libraries adding rocm support because it then ends up targeting such a sliver (and shifting sliver) at that of the market.

Post reply on HN