Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Use your Nvidia GPU's VRAM as swap space on Linux

github.com

Post reply on HN
Print view View original

41–50 of 142 posts

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#41

Post by strictnein »

giancarlostoro wrote: post #29
dragontamer wrote: post #10

Remember how 16GBs used to be an enterprise level database mainframe? Well, GPUs also have stupid amounts of compute on them. I have to imagine that there is some kind of database format that's useful with GPU compute attached. Since the data is already in VRAM, the GPU can sort, join, or otherwise manipulate data as needed.

Can we somehow make them work with 1 TB PCIes so we can churn through way more data?

You are able to use GPU Direct Storage to communicate between the GPU and PCIE storage devices. It's nice, but it's not typically as performant as one would like, in comparison to the onboard memory.

https://docs.nvidia.com/gpudirect-storage/

https://github.com/microsoft/DirectStorage/tree/main

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#42

Post by dragontamer »

giancarlostoro wrote: post #29
dragontamer wrote: post #10

Remember how 16GBs used to be an enterprise level database mainframe? Well, GPUs also have stupid amounts of compute on them. I have to imagine that there is some kind of database format that's useful with GPU compute attached. Since the data is already in VRAM, the GPU can sort, join, or otherwise manipulate data as needed.

Can we somehow make them work with 1 TB PCIes so we can churn through way more data?

Have you heard of the "Radeon Pro SSG" ??

It must have failed because I never heard of an update to this GPU. But AMD definitely made a GPU with 4x NVMe SSDs attached to the GPU.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#43

Post by Teknoman117 »

mmastrac wrote: post #38

I seriously looked at this as a way to improve the RAM situation in a QNAP 2U unit that I was having trouble sourcing RAM for. It's somewhat annoying that legit memory-over-PCIe is gated on PCIe5 and chipset support. In the end I just had to bite the bullet and take a gamble on finding ECC DDR4 RAM that would work with the ancient AMD chipset... This particular implementation seems to be running over too many layers…

Memory on an expansion card isn't gated on PCIe 5, it's gated on CXL support. CXL and PCIe use the same electrical/physical layer but the protocol is very different.

The problem with putting (system) RAM on a PCIe card is that PCIe is not a cache-coherent interconnect. If you have a cache line that resides on your GPU sitting inside your processor's cache a remote modification to that memory by either the GPU, another CPU core or some other PCIe device with NOT invalidate the CPU cache line. You also have the fun situation that if it's modified on both ends simultaneously the resulting state will be non-deterministic.

Device drivers have to be very careful about synchronization when accessing memory-like areas on PCIe. CXL adds a cache coherency protocol among other things, so that invalidations and snoops can be exchanged over the interconnect.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#45

Post by sidewndr46 »

kccqzy wrote: post #28
dlt713705 wrote: post #26

Does anyone these days really use swap for anything than S4 suspend ?

https://news.ycombinator.com/item?id=40697318 This HN comment and the linked post brought up a lot of good points. The main takeaway is that swap should primarily be considered a mechanism for equality of reclamation, not for emergency extra memory, where equality of reclamation means file-backed pages and anonymous pages are subject to similar criteria for being evicted from physical memory. I used to have zero swap…

I just set swappiness to zero years ago and never looked back.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#46

Post by Teknoman117 »

RachelF wrote: post #17

Nice idea, but something has gone very wrong here: >Sequential throughput: ~1.3 GB/s [on a RTX 3070 Laptop] This RTX 3070 chip is on PCIe 4.0 x16 which should give 64GB/s. The 8GB of GDDR6 is 448GB/s. Swapping to an NVMe drive would be twice as fast, but with higher latency.

Gen 4.0 x16 is 32 GB/s in each direction, but the way this is implemented is not the way you'd go about this if you wanted high performance.

Edit: Their benchmarks are also run using ZRAM, which compresses pages before writing to swap. Not sure what the performance overhead of that is, but it's probably quite a bit.

First of all, it's a userspace program hooking the nbd driver, which is known for being slow. It also uses a bounce buffer in userspace before transferring to the GPU. So when the kernel needs to swap a page, it has to first copy it into a userspace facing buffer. The userspace program that has to wake back up and issue the cuda operation to copy the page into device memory.

nbd also doesn't really do a good job of supporting high queue depth or merging adjacent accesses. So if the kernel is issuing a bunch of 4K page swaps without any coalescing, you're going to end up with at least million kernel/userspace context switches per second just to handle 4 GB/s (4 GB / 4K page), let alone 64 GB/s. And that's just the NBD portion, forget the mess that is the NVIDIA driver. PCIe can move a lot of data, but in order to get anything even resembling the full bandwidth, you have to have use DMA engines with long page lists. Having to set up a transfer for every 4K page over PCIe will not reach full saturation of the bus.

Swapping to NVMe is a very optimized path -> the swapper can submit lists of pages directly to the NVMe driver and the controller can DMA them directly out of RAM, no copies or context switches CPU side at all.

This could probably be improved by migrating to the ublk driver as it might let you avoid the userspace bounce buffer. It'd also be able to have multiple write queues to at least set up CUDA copies in parallel.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#47

Post by kccqzy »

sidewndr46 wrote: post #45
kccqzy wrote: post #28

Earlier quoted context omitted.

https://news.ycombinator.com/item?id=40697318 This HN comment and the linked post brought up a lot of good points. The main takeaway is that swap should primarily be considered a mechanism for equality of reclamation, not for emergency extra memory, where equality of reclamation means file-backed pages and anonymous pages are subject to similar criteria for being evicted from physical memory. I used to have zero swap…

I just set swappiness to zero years ago and never looked back.

That’s like the complete opposite advice. Chris said the lowest recommended swappiness is 1. I have it set to 100.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#49

Post by wang_li »

Teknoman117 wrote: post #43
mmastrac wrote: post #38

I seriously looked at this as a way to improve the RAM situation in a QNAP 2U unit that I was having trouble sourcing RAM for. It's somewhat annoying that legit memory-over-PCIe is gated on PCIe5 and chipset support. In the end I just had to bite the bullet and take a gamble on finding ECC DDR4 RAM that would work with the ancient AMD chipset... This particular implementation seems to be running over too many layers…

Memory on an expansion card isn't gated on PCIe 5, it's gated on CXL support. CXL and PCIe use the same electrical/physical layer but the protocol is very different. The problem with putting (system) RAM on a PCIe card is that PCIe is not a cache-coherent interconnect. If you have a cache line that resides on your GPU sitting inside your processor's cache a remote modification to that memory by either the GPU, anothe…

It’s deterministic. But as the user you don’t know enough to know what was determined.

Re: Use your Nvidia GPU's VRAM as swap space on Linux

#50

Post by tempoponet »

mathisfun123 wrote: post #48
xfalcox wrote: post #7

Given my dev machine has 32GB of RAM and 32GB of VRAM that sits mostly idle when I'm not running AI models, this is not that bad of an idea.

this is the pcmasterrace equivalent of being all upper body and with scrawny legs lol

It's fine for dense models where you need them in VRAM, less so for MoE where you're offloading layers to ram. But 32/32 is pretty good for both in the popular ~30b range right now.
Post reply on HN