Potentially noob question, with Moore's law slowing down, are there enough specializations/hardware modifications available like those mentioned in the paper such that progress in ML will continue to progress rapidly? or will these advancements simply forestall an inevitable asymptote.
It's little more than an educated guess on my part, but I figure there's about two orders of magnitude in improvements in processing speed exploitable with current processes, if a big-budget chip were designed specifically for ML training. GPUs are architecturally not very optimal for the task. You want something like a chip with a huge mesh of small independent cores with their own local storage, quite possibly with…
TPU is the main ML ASIC in use. A major goal of the original TPU design seems to be reducing the number of memory accesses. The other top-end ML device is NVIDIA's GPUs with Tensor Cores. Both of those chips are designed around fast matrix multiplication, which right now seems to be the most important operation in deep learning - see how RNNs have started to fall out of favor to CNN-based networks with attention heads.
The TPU is not faster than NVIDIA's GPUs, but it is cheaper. Right now the future seems to be cheaper ML devices designed to be horizontally scalable.
From the CPU perspective, it appears that the major ML effort is related to vectorizing instructions via advanced instruction sets.
Everyone who creates silicon is focused very heavily on using smaller and smaller numeric types - float16 is standard and there is work being done for even smaller int based work.
I haven't seen any analog-based ML devices in use. Can you share an example? Is there even a way to approximate the results of a matmul using analog devices?
It's impossible to guess how much more speed we can get with current approaches, but everything from silicon to networking stack to libraries to network architectures are in their infancy so I would expect dramatic improvements in performance on a regular basis (but not as regular as other areas of software because silicon development is slow)