31–40 of 50 posts
A GPU seems overkill when the bottleneck is the I/O.
In systems performance I would advise to never think of any workload as unidimensional (ie: Any file system optimization can either improve IO latency or be useless) Issuing individual truncates of 1B files can be just as much of a CPU problem then an IO one for example.
Interesting they would discuss system call overhead of opening a file, reading from it and closing it. Seems like in almost all cases the open and close calls would be overwhelmed by the other operations.
Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work. edit: blind as a bat, says so right in the paper of course: PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA…
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Once you got that then the CPU is just the orchesterator, and wouldn't necessarily need to be so beefy.
Given that PCIe allows data to be piped directly from one device to another without going through the host CPU[1][2], I guess it might make sense to just have the GPU read blocks straight from the NVMe (or even NVMe-of[3]) rather than having the CPU do a lot of work. edit: blind as a bat, says so right in the paper of course: PMem is mapped directly to the GPU, and NVMe memory is accessed via Peer to Peer-DMA (P2PDMA…
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Also, Optane was like $4 per GB, so a moderately-sized drive, like 256GB, is already above $1000.
Earlier quoted context omitted.
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Didn't they stop making Optane? :( Also, Optane was like $4 per GB, so a moderately-sized drive, like 256GB, is already above $1000.
Earlier quoted context omitted.
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Sure, but my point was that it should be quite possible to get regular NVMes working. Once you got that then the CPU is just the orchesterator, and wouldn't necessarily need to be so beefy.
The PS5 and Xbox both have GPU-access of NVMe Flash.
------
So you are right. But what you are talking about happened like 5 years ago.
EDIT: https://devblogs.microsoft.com/directx/directstorage-develop...
Looks like 3 years ago for Win10. But I feel like I heard it sooner than that as NVidia or AMD specific API calls.
Earlier quoted context omitted.
I'm not sure they're actually doing NVMe yet; using Optane PMem is a bit of a cheat so that accessing storage is just plain memory reads and writes over PCIe. Implementing an NVMe device driver to set up and interact with command queues would be an extra layer of complexity that I think they left for future work.
Didn't they stop making Optane? :( Also, Optane was like $4 per GB, so a moderately-sized drive, like 256GB, is already above $1000.
But this work used the Optane DC Persistent Memory DIMMs that only work with certain Intel server CPUs. I'm not sure what the typical price people actually paid for those was, but it probably was not actually more expensive than DRAM.
While it is not a 1:1 comparison there has been a driver for windows that allows the creation of a ram drive from vram for NVIDIA cards. >GpuRamDrive >Create a virtual drive backed by GPU RAM. https://github.com/prsyahmi/GpuRamDrive Fork with AMD support: https://github.com/brzz/GpuRamDrive/ Fork that has fixes and support for other cards and additional features: https://github.com/Ado77/GpuRamDrive
While it is not a 1:1 comparison there has been a driver for windows that allows the creation of a ram drive from vram for NVIDIA cards. >GpuRamDrive >Create a virtual drive backed by GPU RAM. https://github.com/prsyahmi/GpuRamDrive Fork with AMD support: https://github.com/brzz/GpuRamDrive/ Fork that has fixes and support for other cards and additional features: https://github.com/Ado77/GpuRamDrive
seq1m 2205 2190 q1t1
rndq32 41.31 38.77
rnd q1t1 34.70 32.80
To be honest i didn't know what to expect, aside for a very high reading and writing speed. I was a bit disappointed in seeing random reading and writing were so slow, the only use i could think about would be having photosets or things like that over there, and then saving the session on ssd when closing the program, but it is easily solved by using a newer nvme ssd