Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

gitlab.com

Post reply on HN
Print view View original

91–100 of 149 posts

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#91

Post by HighGoldstein »

M95D wrote: post #90
wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

> A connector of any kind reduces signal quality.

Like the M.2 connector?

> Data lines need to be longer

Like the data lines going all the way to an on-motherboard storage device?

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#93

Post by robotswantdata »

0xbadcafebee wrote: post #18

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may sho…

12 channel ddr5 5600 ECC is around 500gbs which in real world works very well for large MoE

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#94

Post by wewewedxfgdf »

VHRanger wrote: post #88
wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

GDDR7x doesn't come in dimm factor? In general soldered ram seems to get much higher bandwidth than removeable ram. See ryzen AI Max vs 9950x max ram throughputfor example

We are talking here about slower ram to augment.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#95

Post by wewewedxfgdf »

timmmmmmay wrote: post #89
wewewedxfgdf wrote: post #85

Why don't they just put ram slots on the card so you can augment the fast ram

connectors are bad for signal integrity and GDDR is particularly picky about this

We're talking about ordinary RAM to augment, like a cache.

Not as GPU VRAM expansion.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#96

Post by literalAardvark »

dataflow wrote: post #62
jbverschoor wrote: post #60

Earlier quoted context omitted.

Not true for unified systems. And for strix halo you need to dedicate the amount which is annoying. You’re basically stating that swapping is also a bad idea. And to take it further, any memory or storage is a bad idea because there’s L1 cache/SRAM which is faster then the rest

> You’re basically stating that swapping is also a bad idea. Is that a crazy thing to say? I can't recall the last time I was grateful for swap; it might've been before 2010.

If you've used any unreserved VM ever you're grateful for swapping.

Somewhat indirectly but still.

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#97

Post by literalAardvark »

HighGoldstein wrote: post #91
M95D wrote: post #90

Earlier quoted context omitted.

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

> A connector of any kind reduces signal quality. Like the M.2 connector? > Data lines need to be longer Like the data lines going all the way to an on-motherboard storage device?

Soldered stuff is still dramatically better than the M2 connector (than any connector really). You've never wondered why RAM doesn't use PCI Express?

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#98

Post by zbentley »

HighGoldstein wrote: post #91
M95D wrote: post #90

Earlier quoted context omitted.

Speed and reliability. A connector of any kind reduces signal quality. Data lines need to be longer, because the memory slot won't fit under the radiator where the memory chips are now, and that adds even more electrical interference and degrades signal. Also, we had memory slots on '90s cards. They were extremely expensive and proprietary. Ever saw a Matrox VRAM card? I never did.

> A connector of any kind reduces signal quality. Like the M.2 connector? > Data lines need to be longer Like the data lines going all the way to an on-motherboard storage device?

> Like the M.2 connector?

Yes, though likely something with a higher pin count since memory access is more likely to be random and can be parallel versus block storage.

> Like the data lines going all the way to an on-motherboard storage device?

Yes. Why would a GPU manufacturer/packager take on that cost, if it’s presently served well enough for most people by offloading it onto other parts of the system?

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#99

Post by ninjagoo »

This is awesome! Normally, offloading layers to the CPU RAM means that the compute for those layers occurs on the CPU instead of the GPU, generally speaking. The CPU is orders of magnitude slower than the GPU.

With this approach the compute occurs on the GPU, with the tradeoff that layers in RAM have to be moved back-and-forth through PCI-DMA. It seems to me that this should offer a speedup vs compute split between GPU and CPU. The amount of speedup will depend on how many layers would have been on CPU compute, minus the reduction due to moving those layers between RAM and the GPU.

What's slower? Compute on the CPU or moving data from RAM to GPU through PCI-DMA?

Re: Nvidia greenboost: transparently extend GPU VRAM using system RAM/NVMe

#100

Post by adrian_b »

0xbadcafebee wrote: post #18

You can already do this with some GPU drivers: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash amdttm.pages_limit=5242880 ttm.pages_limit=5242880" One downside is your kernel isn't going to reserve that memory away from userland. You will still see all the memory at system level as "free". As the GPU driver starts using it, other apps/the OS will try to use the "free" memory, not knowing how much of it is in use (it may sho…

With discrete GPUs, using system RAM is slow not due to mem bandwidth, but due to PCIe bandwidth, which is the bottleneck.

For example, 16x PCIe 4.0: 256 Gb/s, 16x PCIe 5.0: 512 Gb/s, while 2x DDR5-6400 DIMMs: 819 Gb/s. The actual throughput is lower for both PCIe and DDR5, due to communication overhead.

On server/workstation motherboards which may have 4, 8 or 12 DIMMs instead of 2, the ratio between memory bandwidth and PCIe bandwidth becomes proportionally higher, so the memory throughput achievable by the GPU becomes a very small fraction of the system memory bandwidth.

Post reply on HN