Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Full-scale file system acceleration on GPU [pdf]

dl.gi.de

Post reply on HN
Print view View original

41–50 of 50 posts

Re: Full-scale file system acceleration on GPU [pdf]

#41

Post by loeg »

maxcoder4 wrote: post #23
loeg wrote: post #14

Earlier quoted context omitted.

It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. I think the main benefit here is not having to do memory copies through the CPU, which frees up memory bandwidth for other things.

There are plenty of cases where you can't just change the file layout. And the GPU filesystem is being implemented by someone else, so the choice is: migrate your data to another filesystem OR fix the data-in-files layout, even though the files may come from completely different source than your application, the layout may be a standard or other applications may depend on it, or you can't easily change it for another…

If you can get the data into the GPU-native filesystem, you can change the data layout at least as easily. The point is there is some sort of data ingestion pipeline involved.

Re: Full-scale file system acceleration on GPU [pdf]

#42

Post by loeg »

eru wrote: post #25
loeg wrote: post #14

Earlier quoted context omitted.

It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. I think the main benefit here is not having to do memory copies through the CPU, which frees up memory bandwidth for other things.

> It seems more straightforward to fix your data-in-files layout than to implement a novel in-GPU filesystem, though. You can improve file-open overhead in conventional filesystems, too. Including the FUSE one I was working on.

Sure!

Re: Full-scale file system acceleration on GPU [pdf]

#43

Post by solardev »

multimind wrote: post #15

A friend of mine used to work for a GPU database startup as an integration engineer. He got frustrated because GPU drivers ( not just AMD but also Nvidia ) are intrinsically unstable and not designed for long flawless runs. If a few bits have a wrong value in a deep neural network or a pixel is wrong in a game, it does not matter much. In databases ( or file systems for that matter ) it does mean everything! It is ha…

Could you use some sort of RAID array of GPUs to compensate...?

Re: Full-scale file system acceleration on GPU [pdf]

#45

Post by _kdave »

I'm glad that research papers don't start with "we've analyzed linux kernel 2.6.18 sources (because this is what we had on our lab machines) and determined that ext3 is the best filesystem for our research purpose and now present you with a novel idea of using high-tech device on that". The paper acknowledges modern features, takes design from other filesystems (mentioned BTRFS and tree structures). Overall the idea is interesting and promising.

Re: Full-scale file system acceleration on GPU [pdf]

#47

Post by az226 »

magicalhippo wrote: post #3

Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work. edit: blind as a bat, says so right in the paper of course: PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA…

For GPUs where Nvidia has turned off P2P, can RAM or NVMe drives be used for emulating P2P? Let’s assume you have a RAID AIC with 4 or 8 high speed SSDs. Could you make 3 3090s work as well as 3 A5000 RTX for training a model?

Re: Full-scale file system acceleration on GPU [pdf]

#48

Post by nonplus »

solardev wrote: post #43
multimind wrote: post #15

A friend of mine used to work for a GPU database startup as an integration engineer. He got frustrated because GPU drivers ( not just AMD but also Nvidia ) are intrinsically unstable and not designed for long flawless runs. If a few bits have a wrong value in a deep neural network or a pixel is wrong in a game, it does not matter much. In databases ( or file systems for that matter ) it does mean everything! It is ha…

Could you use some sort of RAID array of GPUs to compensate...?

nvidia-smi exposes all cards, so you could run the same workload on multiple cards. This (likely) won't solve the problem of certain failure modes being intrinsic to the work being completed/compute environment. I would speculate some of those aggressive failure modes would present themselves across all the hardware.

Maybe someone could run workloads across CUDA and ZLUDA (Nvidia, and other hardware), but really we just might need more reliability to efficiently and reliability run a file system across disparate GPU hardware.

Re: Full-scale file system acceleration on GPU [pdf]

#49

Post by riedel »

afr0ck wrote: post #20

I didn't fully read the paper, but few questions come into mind. 1) How does this work differ from Mark Silberstein's GPUfs from 2014 [1]? 2) Does this work assume the storage device is only accessed by the GPU? Otherwise, how do you guarantee consistency when multiple processes can map, read and write the same files? You mention POSIX. POSIX has MAP_SHARED. How is this situation handled? 3) Related to (2), on the de…

> 1) How does this work differ from Mark Silberstein's GPUfs from 2014 [1]?

Just quoting the paper:

>Using GPUfs, Silberstein et al . [ 24] demonstrate that offering a library interface to CPU FS eases access to storage for GPU programmers, but GPUfs only calls a CPU-side file system. GPU4FS offers a similar interface to GPUfs, but runs the file system on the GPU.

Re: Full-scale file system acceleration on GPU [pdf]

#50

Post by afr0ck »

riedel wrote: post #49
afr0ck wrote: post #20

I didn't fully read the paper, but few questions come into mind. 1) How does this work differ from Mark Silberstein's GPUfs from 2014 [1]? 2) Does this work assume the storage device is only accessed by the GPU? Otherwise, how do you guarantee consistency when multiple processes can map, read and write the same files? You mention POSIX. POSIX has MAP_SHARED. How is this situation handled? 3) Related to (2), on the de…

> 1) How does this work differ from Mark Silberstein's GPUfs from 2014 [1]? Just quoting the paper: >Using GPUfs, Silberstein et al . [ 24] demonstrate that offering a library interface to CPU FS eases access to storage for GPU programmers, but GPUfs only calls a CPU-side file system. GPU4FS offers a similar interface to GPUfs, but runs the file system on the GPU.

Thanks for the quote!

In this case, it is indeed novel to run the logic of the filesystem on the GPU itself. It's definitely worth the investigation!

Post reply on HN