From what I gather online, intel is entering into an exestential crisis as homegrown chips and GPUs are replacing CPUs in the battle for tomorrow's data center. All of the interesting research from Google and others all have one thing in common: a custom asic processor.
The ASIC processors Google use are for inference , not training. Having efficient inference is great, but training is what uses up most of the compute power. The battle between GPUs and CPUs in the datacenter is real, but I wouldn't count out regular CPUs yet. Even with GPU-ready frameworks like Tensorflow, Theano, and Pytorch, it still requires a fair amount of domain expertise to get good performance out of GPUs. T…
As to whether the TPU can perform efficient training or whether it chokes like current FPGAs due to insufficient memory bandwidth is a different question. That would be my guess but I'd love to be proven wrong.