> The bottleneck for GPU today is the amount of memory on board and not compute. Especially with large size models.
Well... everything "depends on the model". Some models will be GPU-compute limited, others will be bandwidth-limited, and others will be capacity limited.
> In addition, the problem with deep learning, in general, is the seq nature of the alg. I.e it is parallel within the layer, but not between layers. And for multi gpu setup, again it is the communication link between the GPUs.
You should increase the size of the model to increase parallelism. Any problem has sequential-bits and parallel bits, so we can't stop the sequential nature of problems.
But the idea is to think NOT in terms of Ahmdal's law, but instead in terms of Gustafson's law. When you have a computer that's twice-as-parallel, you need to double the work done.
You can't reasonably expect things to get faster-and-faster (Ahmdal's law). Instead, you load up more-and-more work the wider-and-wider machines get.
In the case of neural nets, instead of doing 128x128x5 sized kernels, you upgrade to 256x256x5 sized kernels as GPUs get thicker.
Moore's law was never about the "speed of computers", but instead about "the number of transistors". As such, its Gustafson's law that best scales with Moore's law. We've got another 5 to 10 years left in Moore's law by the way: denser memory, denser GPU compute, 5nm-class chips (probably networked as a bunch of chiplets).
We will have more CPU / GPU power, as well as more RAM density, in the future. The question is how to organize our code so that we'll be ready 5 years from now for the next-gen of hardware.