Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

gitlab.com

Post reply on HN
Print view View original

41–50 of 149 posts

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#41

Post by alexeldeib »

Havoc wrote: post #24

> The best strategy is to shrink the model until it fits — either with EXL3 quantization or ModelOpt PTQ — and use GreenBoost's DDR4 pool for KV cache only. Does this make sense? I'd have thought the KV is guaranteed to be used 100% of the time while say in a MoE the same can't be said of the weights. Though I suppose if you're shooting for huge context then having that allocation go into ram makes sense specially wh…

KV cache is, well, a cache that can fill up and trigger eviction. You require enough space to execute at least 1 fwd pass of 1 request at your context length. KV cache hits reduce TTFT by avoiding prefill. You don’t get to skip decode.

MoE is kinda related in terms of lower usage requirements vs a dense model of same total param size, but I think your mental model is a bit off.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#43

Post by timnetworks »

smallnamespace wrote: post #36
jmward01 wrote: post #31

Earlier quoted context omitted.

The point is not how fast it is now. The point is that this opens new possibilities that can be built on. Potentially models that are trained with slightly different architectures to optimize to this use case. Possibly others come to improve this path. Possibly HW manufacturers make a few small adjustments that remove bottlenecks. Who knows, the next person may combine CPU compute with this mem sharing to get another…

It’s architecturally not a good approach. System RAM is much slower so you should put data that doesn’t need to be used often on it. That knowledge is at the application layer. Adding a CUDA shim makes system RAM appear like VRAM, which gets things to run, but it will never run very well. The benchmarks at the bottom mention memory tiering and manually controlling where things go, but if your application already does…

Some people are not concerned with having it run the fastest, just having it run at all may be enough.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#44

Post by midnitewarrior »

smallnamespace wrote: post #36
jmward01 wrote: post #31

Earlier quoted context omitted.

The point is not how fast it is now. The point is that this opens new possibilities that can be built on. Potentially models that are trained with slightly different architectures to optimize to this use case. Possibly others come to improve this path. Possibly HW manufacturers make a few small adjustments that remove bottlenecks. Who knows, the next person may combine CPU compute with this mem sharing to get another…

It’s architecturally not a good approach. System RAM is much slower so you should put data that doesn’t need to be used often on it. That knowledge is at the application layer. Adding a CUDA shim makes system RAM appear like VRAM, which gets things to run, but it will never run very well. The benchmarks at the bottom mention memory tiering and manually controlling where things go, but if your application already does…

[dead]

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#45

Post by nl »

3abiton wrote: post #29
nl wrote: post #25

Earlier quoted context omitted.

The Ryzen AI CPU/GPUs (Ryzan AI 395+ etc) seem to have increasing support - https://lemonade-server.ai/ now has support for the NPU as well as the combined CPU/GPU (which I guess is a APU but is different to the G series of APUs I think?) But I'm always interested in first hand experiences of how good is it really - I'm pretty cynical about the idea that AMD actually knows what it takes to build good software end-to-…

I also have one, and indeed support is very much frictionless now compared to a year ago. But again, not thanks to AMD, as initially it was purely community driven. Strix halo was not even supported by ROCm (officially), and we had to deal with therock images, then donato made the toolbox, and then lemonade came through. I am really surprised how AMD approached this. They made big promises, they threw the hardware ou…

I believe Lemonade is the AMD team right?

But yes I agree with you about their lack of prioritization for software!

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#46

Post by rvz »

SV_BubbleTime wrote: post #33
Insanity wrote: post #30

Extend your VRAM using RAM, then extend your RAM using Swap.

If you are doing video models, this is an excellent way to murder your SSD. Do not put swap on an SSD you care about at all.

> Do not put swap on an SSD you care about at all.

This.

Many people rediscovering what the purpose of swap files are, but will still find a way to abuse it without knowing that they are actually destroying their SSD.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#48

Post by cma »

NooneAtAll3 wrote: post #47

nvidia failed to provide gpu with actually meaningful amount of vram and instead of improving the actual product, it decided to "solve the problem in software" I expect this greenboost to fall and burn, honestly...

> it decided to "solve the problem in software"

This isn't made by nvidia

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#50

Post by lelanthran »

0xbadcafebee wrote: post #18

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may sho…

> any case, loading a gigantic model just to use system RAM is absurdly slow (due to mem bandwidth), like 1-5 t/s, so it's not practical. It'd take a whole day to process one 86k token reques

So don't use it for large requests. Ideal for when you just want to categorise things, for example, "does this task need a shell" or "bucket this email into one of help request, bill due or personal comms".

Post reply on HN