For the curious, that's a plaque on the side if the rack showing the Go board at the end of AlphaGo vs Lee Sedol Game 3, at the moment Lee Sedol resigned and AlphaGo won the tournament (of five games).
Google supercharges machine learning tasks with TPU custom chip
71–80 of 283 posts
Re: Google supercharges machine learning tasks with TPU custom chip
#72I'm happy to hear that this is finally public so I can actually talk about the work I did when I was at Google :-). I'm a bit surprised they announced this, though. When I was there, there was this pervasive attitude that if "we" had some kind of advantage over the outside world, we shouldn't talk about it lest other people get the same idea. To be clear, I think that's pretty bad for the world and I really wished th…
Google has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had. My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo…
> By revealing that AlphaGo was based on this hardware
Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.
Re: Google supercharges machine learning tasks with TPU custom chip
#73Perf/W, the official metric of slow but efficient processors. How many times must we go down this road?
Let's see this sucker train AlexNet...
Re: Google supercharges machine learning tasks with TPU custom chip
#74Now this is really interesting. I've been asking myself why this hadn't happened before. Its been all software, software, software for the last decade or so. But now I get it. We are at a point in time where it makes sense to adjust the hardware to the software. Funny how things work. It used to be the other way around.
This is known as the Wheel of Reincarnation. Functionality moves to special-purpose hardware then back to software, and the cycle repeats. (The computer term is from 1968 so this has been happening for a long time.) http://www.catb.org/jargon/html/W/wheel-of-reincarnation.htm...
Wonder what it will do the the industry term Full Stack Developer. Will people who call themselves that will now need to know about chip design?
OT: Cool blog! :)
Re: Google supercharges machine learning tasks with TPU custom chip
#753 generations ahead of moore law??? I really wonder how they are accomplishing this beyond implementing the kernels in hardware. I suspect they are using specialized memory and an extremely wide architecture. Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LO…
From the article: "TPU is tailored to machine learning applications, allowing the chip to be more tolerant of reduced computational precision, which means it requires fewer transistors per operation."
Do you reckon that means it's using small floats?
Re: Google supercharges machine learning tasks with TPU custom chip
#763 generations ahead of moore law??? I really wonder how they are accomplishing this beyond implementing the kernels in hardware. I suspect they are using specialized memory and an extremely wide architecture. Sounds they also used this for AlphaGo. I wonder how badly we were off on AlphaGo's power estimates. Seems everyone assumed they were using GPU's, sounds like they were not. At least partially. I would really LO…
It seems entirely reasonable to me, and there's good historical precedent for it. Let's look at SHA256 hashing as an example. The maximum number of hashes that the best GPU around can do is around 1 GHash/s. However, for the same cost, of around $600, specialized hardware can do around 5 THash/s. That's about five thousand times the performance/price. There's no reason that hardware that is super-specific to neural network computation can't similarly have large gains.
Link to the specialized hardware for SHA256 hashing: http://www.amazon.com/Antminer-~4-73TH-25W-Bitcoin-Miner/dp/...
Re: Google supercharges machine learning tasks with TPU custom chip
#77Earlier quoted context omitted.
But isn't 3 generations ahead just 8x? Which doesn't sound at all unreasonable for a custom hardware.
This is about right! 64-bit IEEE fp -> 16-bit IEEE-style fp[0] is a 4x bit size reduction, and multiplication is O(n^2) is silicon transistor count. [0] If google is smart, they'd ditch +/- infinity and if they were ballsy, they'd ditch zero in their FP implementation.
I doubt FP hardware size is the limiting factor in their implementation (it's not in GPUs and definitely not in high perf CPUs). More likely they came up with higher level architectural tricks that let them specialize for machine learning (i.e. taking better advantage of locality in the application, etc).
Re: Google supercharges machine learning tasks with TPU custom chip
#78Earlier quoted context omitted.
Google has always been strategic about announcing what it was doing, even since the early days (I used to work there too). Think about the impact the first MapReduce and GFS/BigTable papers had. My guess as to why they're announcing the TPU is that they are feeling the pressure from Facebook and other AI labs, and want to reinforce their reputation as being the best place to do AI research. By revealing that AlphaGo…
> By revealing that AlphaGo was based on this hardware Interesting, as the nature/science paper made no mention of this, it was exclusively trained on GPUs.
Training and inference are not the same thing.
Re: Google supercharges machine learning tasks with TPU custom chip
#79This is huge. If they really do offer such a perf/watt advantage, they're serious trouble for NVIDIA. Google is one of only a handful of companies with the upfront cash to make a move like this. I hope we can at least see some white papers soon about the architecture--I wonder how programmable it is.
I don't see any mention of offering these chips for sale. You can rent them it seems via cloud offerings & that's it.
Sure, but that's the deal. I'll buy the latest nVidia 1080 card as soon as I can but renting these custom chips per minute would be a way better option for me.
Re: Google supercharges machine learning tasks with TPU custom chip
#80I think this shows a fundamental difference between Amazon (AWS) and Google Cloud. AWSs offerings seem fairly vanilla and boring. Google are offering more and more really useful stuff: - cloud machine learning - custom hardware - live migration of hosts without downtime - Cold storage with access in seconds - bigquery - dataflow
Vanilla? Boring?
I read "Vanilla" and "Boring" as "Horray, I don't have to spend time rewriting all this complicated code I already have!"
If I'm just dipping my toes into (say) Caffe or Theano, I don't have to rewrite it from scratch.
That is a huge advantage---not a disadvantage!---of AWS over google.