Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

gitlab.com

Post reply on HN
Print view View original

141–149 of 149 posts

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#141

Post by M95D »

whalesalad wrote: post #133
M95D wrote: post #90

Earlier quoted context omitted.

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

I am hoping that we seriously evolve the ATX standard to allow for a socketed GPU board that can also enable user replaceable memory. Seeing an enormous GPU that is larger than the motherboard itself hanging from a PCI slot feels like horse and buggy shit. I'm imaging two boards back-to-back connected by a central high bandwidth bus (which could also do power delivery) that would allow one side of the case to be for…

Your solution only allows for one GPU, maybe two if the motherboard is really huge, and it doesn't really solve the slotted VRAM problem.

PCI is (was) allowed to be even longer. Old AT and ATX cases had a slotted support bracket to hold the far end of the PCI cards. See how an Adaptec 2400A looks like.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#142

Post by Melatonic »

adrian_b wrote: post #100
0xbadcafebee wrote: post #18

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may sho…

With discrete GPUs, using system RAM is slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck. For example, 16x PCIe 4.0: 256 Gb/s, 16x PCIe 5.0: 512 Gb/s, while 2x DDR5-6400 DIMMs: 819 Gb/s. The actual throughput is lower for both PCIe and DDR5, due to communication overhead. On server/workstation motherboards which may have 4, 8 or 12 DIMMs instead of 2, the ratio between memory bandwidt…

Maybe then this is a forward thinking feature for when we (maybe) get improved GPU hardware slots?

edit: Are you sure PCI-E is even that fast? Looking at the chart on Wikipedia (did not research further - so grain of salt here) shows much lower throughput

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#143

Post by 0xbadcafebee »

stuaxo wrote: post #111
Tsiklon wrote: post #106

Earlier quoted context omitted.

Strix Halo’s unified setup is pretty cool. In systems with 128GB of memory, in BIOS set the dedicated GPU memory to the smallest permitted and the Drivers will use the whole main memory pool appropriately in Linux and Windows

Does this work on the open source amdgpu drivers ? I've been a bit too busy to turn mine on for a while.

The OSS AMDGPU drivers by default allocate a fixed percentage of system RAM for GTT (up to 75%), they do not automatically use the entire system memory. You can override this with the kernel options I posted in my original comment, but as I mentioned, there are some serious negative consequences. You also may need to disable IOMMU or use PT mode. Personally I have had a lot of crashes as a result of this stuff, so I went back to the defaults and just don't run big models.

The biggest factor of whether AMD GPUs on Linux are a PITA or not is ROCm. Strix Halo is supported in ROCm 6.x, so it should be supported on most platforms (I haven't tested it tho). ROCm 7.x is supposed to be better but not all apps support it yet.

AMD, if you're reading this, please hire more SWEs. Nvidia will continue to dominate until you beat them at software.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#144

Post by zargon »

HighGoldstein wrote: post #91
M95D wrote: post #90

Earlier quoted context omitted.

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

> A connector of any kind reduces signal quality. Like the M.2 connector? > Data lines need to be longer Like the data lines going all the way to an on-motherboard storage device?

Yes and yes. NVMe storage is very slow, so it can get away with such things.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#145

Post by VHRanger »

nic547 wrote: post #140
VHRanger wrote: post #128

Earlier quoted context omitted.

It's really hard to push DDR5 past 6000MT/s on 4+ DIMMs it seems. I had to get everything top spec to fit 4 channels of 6000MT/s on my 9950x (asus proArt motherboard and the top tier trident neo RAM sticks) -- otherwise it's reportedly unstable.

9950X is dual channel, running 4 DIMMs runs them interleaved, with two DIMMs sharing one physical connection, impacting signal integrity severely. AFAIK this has gotten worse with DDR5 to the point that it's generally recommended to avoid 4 DIMMs unless you really can't get enough RAM otherwise. For maximum bandwidth you need to avoid interleaving. Strix Halo simply has more memory controllers. Threadrippers are also…

And I presume that doubled price is before you look at the workstation class motherboards, which also tend to be much more expensive.

Thanks for the info on the hardware quirks, useful to know!

We seem to be arriving at a cambrian explosion of viable hardware these days between ARM and x86, soldered vs DIMM, etc.

It's refreshing coming from 20 years of x86 being all that matters.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#146

Post by angry_octet »

Rafuino wrote: post #122
angry_octet wrote: post #104

I have a system with an ungodly amount of Optane memory and I'm hoping this will work.

What do you have? I've got a 905P and a 900P and am already using these in LM Studio by putting all models there and extending system memory with more scratch space... Not sure if I need to do anything differently with this since LM Studio already enabled it I think

The DIMM version, PMEM 200 modules. Appears as /dev/dax and you can mmap it. Could also use as a virtual filesystem, or using RAM as a cache for pmem, but sceptical of that.

Is the virtual memory system is a good way to expand memory for inference, via having it directly manage a non uniform pool? I haven't seen any research on that.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#147

Post by lostmsu »

zargon wrote: post #130
adrian_b wrote: post #101

Earlier quoted context omitted.

You mean 500 GB/s, not Gb/s (actually 537 GB/s). Unfortunately that does not matter. Even in a cheap desktop motherboard the memory bandwidth is higher than of 16-lane PCIe 5.0. Therefore the memory bandwidth available to a discrete GPU is determined by its PCIe slot, not by the system memory. If you install multiple GPUs, in many MBs that will halve the bandwidth of the PCIe slots, for an even lower memory throughpu…

> in many MBs that will halve the bandwidth of the PCIe slots Not on boards that have 12 channels of DDR5. But yeah, squeezing an LLM from RAM through the PCIe bus is silly. I would expect it would be faster to just run a portion of the model on the CPU in llama.cop fashion.

llama.cpp afaik does not run portion of the model on the CPU. --cpu-moe just offloads weights to RAM, but they are still loaded to GPU for compute.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#149

Post by itsTyrion »

svnt wrote: post #15
yjtpesesu2 wrote: post #14

How does this differ from anything llama.cpp offers, regarding offloading layers? The repo consistently refers to "DDR4". Is there a reason DDR5 won't work with this?

The readme opens with this: > I have an RTX 5070 with 12 GB VRAM and I wanted to run glm-4.7-flash:q8_0, which is a 31.8 GB model. The standard options are: > Offload layers to CPU — works, but drops token/s by 5–10× because CPU RAM has no CUDA coherence. You end up waiting. Use a smaller quantization — you lose quality. At q4_0 the model is noticeably worse on reasoning tasks. > Buy a bigger GPU — not realistic for…

> "wanted to run glm-4.7-flash:q8_0" > q8_0

a well made (as in, unsloth) smaller quant will help a good amount here, without a notable reduction in performance or increase in perplexity

Post reply on HN