Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

gitlab.com

Post reply on HN
Print view View original

111–120 of 149 posts

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#111

Post by stuaxo »

Tsiklon wrote: post #106
jbverschoor wrote: post #60

Earlier quoted context omitted.

Not true for unified systems. And for strix halo you need to dedicate the amount which is annoying. You’re basically stating that swapping is also a bad idea. And to take it further, any memory or storage is a bad idea because there’s L1 cache/SRAM which is faster then the rest

Strix Halo’s unified setup is pretty cool. In systems with 128GB of memory, in BIOS set the dedicated GPU memory to the smallest permitted and the Drivers will use the whole main memory pool appropriately in Linux and Windows

Does this work on the open source amdgpu drivers ?

I've been a bit too busy to turn mine on for a while.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#112

Post by natechlin »

shdudns wrote: post #109
zozbot234 wrote: post #102

Earlier quoted context omitted.

> slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. > On server/workstation motherboards ... the memory throughput [to system RAM] achievable by the GPU becomes a very small fraction of the system memory bandwidth. Yes, this is a critical point. It means that this is only realistically useful for prefill, which is compute- and not memory-bandwidth bound.

Sorry, I'm a bit of a noob on llm. What is "prefill"? As opposed to what?

Prefill - module computes KV cache over input toks, up to the last token in your input (the 'prompt'), at which point it can then begin -

Decode - the model chooses a new token to append to the end of the current token list (i.e. it generates a token), then computes the new tokens KVs.

Decode is basically prefill 1 tok -> add 1 tok -> prefill 1 more tok -> ....

but in the initial prefill stage it doesn't need to do generation, since you've provided the toks.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#113

Post by nic547 »

VHRanger wrote: post #88
wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

GDDR7x doesn't come in dimm factor? In general soldered ram seems to get much higher bandwidth than removeable ram. See ryzen AI Max vs 9950x max ram throughputfor example

Strix Halo uses a 256bit memory interface, the normal desktop processors only have a 128bit interface, that's the biggest difference in bandwidth. For more bandwidth you need to go to a Threadripper.

Strix Halo seems to use LPDDR with 8000 MT/s, which is a bit faster than the usual 5600 MT/s-6400 MT/s "normal" DDR5-DIMMs (Albeit (expensive) faster ones seem to exist), so there's a slight edge towards soldered memory (not sure about LPCAMM2 and similar tech).

GDDR7 is a different league, a 5070 Ti also has a 256bit memory interface, but has 896GB/s bandwidth, compared to strix halo with 256GB/s

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#114

Post by ghm2199 »

natechlin wrote: post #112
shdudns wrote: post #109

Earlier quoted context omitted.

Sorry, I'm a bit of a noob on llm. What is "prefill"? As opposed to what?

Prefill - module computes KV cache over input toks, up to the last token in your input (the 'prompt'), at which point it can then begin - Decode - the model chooses a new token to append to the end of the current token list (i.e. it generates a token), then computes the new tokens KVs. Decode is basically prefill 1 tok -> add 1 tok -> prefill 1 more tok -> .... but in the initial prefill stage it doesn't need to do g…

And Incidentally prefill would also be how caching,say, a system prompt saves you some $ for API usage with LLM providers. They only compute the kv cache for the new tokens after the system prompt.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#117

Post by Tsiklon »

stuaxo wrote: post #111
Tsiklon wrote: post #106

Earlier quoted context omitted.

Strix Halo’s unified setup is pretty cool. In systems with 128GB of memory, in BIOS set the dedicated GPU memory to the smallest permitted and the Drivers will use the whole main memory pool appropriately in Linux and Windows

Does this work on the open source amdgpu drivers ? I've been a bit too busy to turn mine on for a while.

I’ve had no issues running GPT-OSS 120b with decent performance on the machine (HP Zbook Ultra G1a). Running on Bluefin/Universal Blue and Windows.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#118

Post by dahart »

dataflow wrote: post #62
jbverschoor wrote: post #60

Earlier quoted context omitted.

Not true for unified systems. And for strix halo you need to dedicate the amount which is annoying. You’re basically stating that swapping is also a bad idea. And to take it further, any memory or storage is a bad idea because there’s L1 cache/SRAM which is faster then the rest

> You’re basically stating that swapping is also a bad idea. Is that a crazy thing to say? I can't recall the last time I was grateful for swap; it might've been before 2010.

Try turning swap off and really find out if you’re not grateful for it. Might be fine if you’re never using all your RAM, but if you are, swap off isn’t fun and you might realize you’ve been unconsciously grateful this whole time. ;) Swap might be important for GPU usage even when not using something like greenboost, since display GPUs sometimes use system RAM to back the GPU VRAM.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#119

Post by dahart »

m-schuetz wrote: post #52
timnetworks wrote: post #43

Earlier quoted context omitted.

Some people are not concerned with having it run the fastest, just having it run at all may be enough.

From my experience, accessing system RAM from the GPU is so slow, it might as well count as "does not work". It's orders of magnitudes faster to memcpy large swaths of memory that you are going to use to the GPU, rather than accessing system mem from a kernel which then takes ages to wait for that small block/page of memory, then waits again for the next small page/block of memory, etc. Latency hiding doesnt work any…

You’re right for some workloads, but not all of them. The same could have been said for disk swap since the beginning though, and people still found it valuable. Disk swapping with spinning drives did used to be multiple orders of magnitude slower than RAM. But it prevented applications or the system from crashing.

Using system memory from the GPU isn’t that bad if your compute is high enough and you don’t transfer that much data. There are commercial applications that support it and only see low 2-digit percentage perf impact and not the multiples you might expect. Plus on Windows on Nvidia hardware, the driver will automatically use system memory if you oversubscribe VRAM, and I believe this was introduced to support running Stable Diffusion on smaller GPUs.

Post reply on HN