Now this is all fun, but has anyone managed to make these mechanisms work with Multicast PCIe ? I really need GPUdirect and StorageDirect to support this, until PCIe catches up to today's (or Blackwell's) NVLink ... around PCIe 12?
Full-scale file system acceleration on GPU [pdf]
21–30 of 50 posts
Re: Full-scale file system acceleration on GPU [pdf]
#22A friend of mine used to work for a GPU database startup as an integration engineer. He got frustrated because GPU drivers ( not just AMD but also Nvidia ) are intrinsically unstable and not designed for long flawless runs. If a few bits have a wrong value in a deep neural network or a pixel is wrong in a game, it does not matter much. In databases ( or file systems for that matter ) it does mean everything! It is ha…
If the game or your training crashes though, it matters a lot. What sort of bugs give you wrong values without crashing, especially driver bugs?.. something is strange here
Re: Full-scale file system acceleration on GPU [pdf]
#23Earlier quoted context omitted.
For lots of small files, that might not be the case. (I worked on a FUSE filesystem that had these issues.)
It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. I think the main benefit here is not having to do memory copies through the CPU, which frees up memory bandwidth for other things.
There are plenty of cases where you can't just change the file layout. And the GPU filesystem is being implemented by someone else, so the choice is: migrate your data to another filesystem OR fix the data-in-files layout, even though the files may come from completely different source than your application, the layout may be a standard or other applications may depend on it, or you can't easily change it for another reason.
Re: Full-scale file system acceleration on GPU [pdf]
#24While it is not a 1:1 comparison there has been a driver for windows that allows the creation of a ram drive from vram for NVIDIA cards.
>GpuRamDrive
>Create a virtual drive backed by GPU RAM.
https://github.com/prsyahmi/GpuRamDrive
Fork with AMD support:
https://github.com/brzz/GpuRamDrive/
Fork that has fixes and support for other cards and additional features:
Re: Full-scale file system acceleration on GPU [pdf]
#25Earlier quoted context omitted.
For lots of small files, that might not be the case. (I worked on a FUSE filesystem that had these issues.)
It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. I think the main benefit here is not having to do memory copies through the CPU, which frees up memory bandwidth for other things.
> It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though.
You can improve file-open overhead in conventional filesystems, too. Including the FUSE one I was working on.
Re: Full-scale file system acceleration on GPU [pdf]
#26Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work. edit: blind as a bat, says so right in the paper of course: PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA…
[deleted]
Re: Full-scale file system acceleration on GPU [pdf]
#27A friend of mine used to work for a GPU database startup as an integration engineer. He got frustrated because GPU drivers ( not just AMD but also Nvidia ) are intrinsically unstable and not designed for long flawless runs. If a few bits have a wrong value in a deep neural network or a pixel is wrong in a game, it does not matter much. In databases ( or file systems for that matter ) it does mean everything! It is ha…
Yeah, I had a lot of nVidia GPUs suddenly disappear mid-training when even nvidia-smi couldn't find them; this was on different systems (Linux) and only a reboot fixed it.
You don't want this kind of thing happening when it is running a filesystem.
Re: Full-scale file system acceleration on GPU [pdf]
#28A GPU seems overkill when the bottleneck is the I/O.
Re: Full-scale file system acceleration on GPU [pdf]
#29A GPU seems overkill when the bottleneck is the I/O.
In systems performance I would advise to never think of any workload as unidimensional (ie: Any file system optimization can either improve IO latency or be useless)
Issuing individual truncates of 1B files can be just as much of a CPU problem then an IO one for example.
Re: Full-scale file system acceleration on GPU [pdf]
#30A friend of mine used to work for a GPU database startup as an integration engineer. He got frustrated because GPU drivers ( not just AMD but also Nvidia ) are intrinsically unstable and not designed for long flawless runs. If a few bits have a wrong value in a deep neural network or a pixel is wrong in a game, it does not matter much. In databases ( or file systems for that matter ) it does mean everything! It is ha…
Yeah, I had a lot of nVidia GPUs suddenly disappear mid-training when even nvidia-smi couldn't find them; this was on different systems (Linux) and only a reboot fixed it. You don't want this kind of thing happening when it is running a filesystem.
Strange. I never had any problem with nvidia GPUs, but I only ever used data center GPU like the V100 (and don't set them up myself). There's a lot of things that go wrong, at least my nvidia GPU always works.