Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

gitlab.com

Post reply on HN
Print view View original

81–90 of 149 posts

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#81

Post by nickjj »

armada651 wrote: post #34

Doesn't Windows already do this by default? I can already run models bigger than my GPU VRAM and it will start using up to 50% of my system RAM as "shared memory". This is on a Desktop PC without a shared memory architecture.

Yep I had a GeForce 750 Ti (2 GB) and I was able to run a ton of things on Windows without any issues at all.

As soon as I switched to Linux I had all sorts of problems on Wayland where as soon as that 2 GB was reached, apps would segfault or act in their own unique ways (opening empty windows) when no GPU memory was available to allocate.

Turns out this is a problem with NVIDIA on Wayland. On X, NVIDIA's drivers act more like Windows. AMD's Linux drivers act more like Windows out of the box on both Wayland and X. System memory gets used when VRAM is full. I know this because I got tired of being unable to use my system after opening 3 browser tabs and a few terminals on Wayland so I bought an AMD RX 480 with 8 GB on eBay. You could say my cost of running Linux on the desktop was $80 + shipping.

A few months ago I wrote a long post going over some of these details at https://nickjanetakis.com/blog/gpu-memory-allocation-bugs-wi.... It even includes videos showing what it's like opening apps both on Wayland and X with that NVIDIA card.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#82

Post by whywhywhywhy »

bigwheels wrote: post #7
holoduke wrote: post #5

The is extremely slow and not useful in my opinion.

Can you elaborate beyond the shallow/superficial dismissal?

If it takes seconds in VRAM it can take tens of minutes running the same thing offloaded to RAM if it hasn't been designed to do it.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#83

Post by shmeeed »

cma wrote: post #48
NooneAtAll3 wrote: post #47

nvidia failed to provide gpu with actually meaningful amount of vram and instead of improving the actual product, it decided to "solve the problem in software" I expect this greenboost to fall and burn, honestly...

> it decided to "solve the problem in software" This isn't made by nvidia

Still kinda true, though. As other commenters have pointed out, their Windows drivers do similar stuff.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#84

Post by lastdong »

whywhywhywhy wrote: post #78
3836293648 wrote: post #37

Earlier quoted context omitted.

I don't think Windows does this, but Ollama does

It's the drivers but it was a relatively recent addition, think it was added when either the 30xx or 40xx series shipped and the lower cards had pitiful VRAM so they enabled it by default so they'd work with all games. Most people who know it does this turns it off because it kicks in too early so if you have 24GB it'll offload to RAM and tank your inference speed when you hit around 22GB use. https://nvidia.custhelp…

Nicely linked!

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#87

Post by segmondy »

yjtpesesu2 wrote: post #14

How does this differ from anything llama.cpp offers, regarding offloading layers? The repo consistently refers to "DDR4". Is there a reason DDR5 won't work with this?

I was wondering the same, but llama.cpp was written to offload to system ram. If this really works, then the advantage could be that one could run transformers / sglang, etc or other tools that don't offload to system ram. However, I want to see the numbers. Perhaps I'll give this a try, but I need a throw away box I could trash if something goes wrong, but have none at the moment.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#88

Post by VHRanger »

wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

GDDR7x doesn't come in dimm factor?

In general soldered ram seems to get much higher bandwidth than removeable ram. See ryzen AI Max vs 9950x max ram throughputfor example

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#90

Post by M95D »

wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal.

Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

Post reply on HN