I hope this brings us closer to widespread external GPUs, where you could use a slower-than-PCIe bus like Thunderbolt 3 or USB 3.1 to upload all assets to the EGPU's SSD during a one-time loading screen.
GpuScan and SSD-To-GPU Direct DMA
21–28 of 28 posts
Re: GpuScan and SSD-To-GPU Direct DMA
#22See also https://developer.nvidia.com/gpudirect and to some extent https://en.wikipedia.org/wiki/NVLink . NVLink is in the Power9 servers Google is using.
Do we have any data that Google is actually using their POWER9 boxes? I always read that as investing in anything not Intel (see also RISC-V), just to be in a better negotiating position with Intel for the vast number of CPUs that they buy.
Re: GpuScan and SSD-To-GPU Direct DMA
#23Earlier quoted context omitted.
I was thinking the same thing, but is SSD to GPU faster than RAM to GPU? In many (not all) cases you buy a tonne of RAM and load your entire dataset into memory once and then iterate over it as necessary. You also lose the flexibility of doing any sort of data modification or augmentation. One domain where your data usually doesn't fit in RAM is image recognition, but often you want to do things like apply random fli…
SSD is probably not as fast as RAM, but it's much much cheaper, in the order of 10x per gigabyte. With SSD-GPU bridge you can have fast access to a multiple TiB training set, on a single machine. Data pre-processing is indeed an issue, but hue adjustment/flipping/cropping could be implemented as Tensorflow operations, on the GPU. Similarly with input decompression - it would either have to be done on GPU, or the data…
As long as the average bandwidth isn't a bottleneck, it's not going to matter - at worst, you're just going to need to prefetch (and due to SSD latency, that's likely optimal regardless).
Re: GpuScan and SSD-To-GPU Direct DMA
#24See also https://developer.nvidia.com/gpudirect and to some extent https://en.wikipedia.org/wiki/NVLink . NVLink is in the Power9 servers Google is using.
Do we have any data that Google is actually using their POWER9 boxes? I always read that as investing in anything not Intel (see also RISC-V), just to be in a better negotiating position with Intel for the vast number of CPUs that they buy.
AFAIK POWER9 is not even out yet.
Re: GpuScan and SSD-To-GPU Direct DMA
#25See also https://developer.nvidia.com/gpudirect and to some extent https://en.wikipedia.org/wiki/NVLink . NVLink is in the Power9 servers Google is using.
Do we have any data that Google is actually using their POWER9 boxes? I always read that as investing in anything not Intel (see also RISC-V), just to be in a better negotiating position with Intel for the vast number of CPUs that they buy.
Allegedly they have Power8 servers in their data centres:
Maire Mahoney, engineering manager at Google and now a director of the OpenPower Foundation, confirmed to The Next Platform that Google does indeed have custom Power8 machines running in its datacenters and that developers can deploy key Google applications onto these platforms if they see fit. Mahoney was not at liberty to say how many Power-based machines are running in Google’s datacenters or what particular workloads were running in production (if any).[1]
It's pretty unclear what that actually means, though.
[1] http://www.nextplatform.com/2016/04/06/inside-future-google-...
Re: GpuScan and SSD-To-GPU Direct DMA
#26Re: GpuScan and SSD-To-GPU Direct DMA
#27This would be incredibly useful for distributed machine learning - imagine a Tensorflow implementation that almost entirely bypasses CPU.
I was thinking the same thing, but is SSD to GPU faster than RAM to GPU? In many (not all) cases you buy a tonne of RAM and load your entire dataset into memory once and then iterate over it as necessary. You also lose the flexibility of doing any sort of data modification or augmentation. One domain where your data usually doesn't fit in RAM is image recognition, but often you want to do things like apply random fli…
RAM-to-GPU is always faster than SSD-to-GPU. It is a solution to help a situation when data size does not fit RAM size (or when user has less budget to purchase enough RAM. In fact, we can purchase Intel SSD 750 (400GB) with 300USD).
Re: GpuScan and SSD-To-GPU Direct DMA
#28Earlier quoted context omitted.
I was thinking the same thing, but is SSD to GPU faster than RAM to GPU? In many (not all) cases you buy a tonne of RAM and load your entire dataset into memory once and then iterate over it as necessary. You also lose the flexibility of doing any sort of data modification or augmentation. One domain where your data usually doesn't fit in RAM is image recognition, but often you want to do things like apply random fli…
RAM-to-GPU is always faster than SSD-to-GPU. It is a solution to help a situation when data size does not fit RAM size (or when user has less budget to purchase enough RAM. In fact, we can purchase Intel SSD 750 (400GB) with 300USD).
For the scenario you're targeting: databases, this makes a tonne of sense, database data regularly exceeds the size of RAM and the operations you want to do on the data are pretty static in the sense that they're the SQL operators.
In deep learning you are usually doing a lot more custom processing and your datasets are usually not as big, such that just buying more RAM is often cost effective.