Do people think that nobody at nVidia has ever heard of specialized deep learning processors? 1. Volta GPUs already have little matmul cores, basically a bunch of little TPUs. 2. The graphics dedicated silicon is an extremely tiny portion of the die, a trivial component (source: Bill Dally, nVidia chief scientist). 3. Memory access power and performance is the bottleneck (even in the TPU paper), and will only continu…
And Intel knew that mobile chips will one day become very popular, too - two decades ago. Much good did that knowledge do to the company. It's not about them being ignorant about it. It's about them making decisions in spite of that knowledge - decisions that Make Sense™ for the advancement and increased profitability of the incumbent cash cow, but they are often contradictory or have a negative impact on new tech in…
I do hope the AI specific TPUs that come out on the future will follow the ARM model instead of being silod into proprietary architectures. Fucking hate the vendor lock in of NVIDIA with CUDA.