Benchmarking Google’s new TPUv2 • Hacker News Board

Re: Benchmarking Google’s new TPUv2

#31

Post by wallflower » Fri, Feb 23, 2018, 5:11 PM UTC

bitL wrote: post #21

Mononokay wrote: post #7

Earlier quoted context omitted.

There's no real need to worry about Google in the long term - nVidia can make back their money solely with their GPUs; Google probably made their expenses back this weekend with searches around the Olympics. It'd be pointless for them to not use their TPUs themselves, and their main product, Adsense, uses ML.

Are you sure about Adsense? Talked to ad pros recently and they all complained Adsense is ancient (still mySQL?) and often broken; doesn't look like Google emphasizes it despite being their cash cow, more like deep state of neglect.

> still mySQL?

The F1 distributed database was developed to move the AdWords business off of MySQL.

https://research.google.com/pubs/pub41344.html

Re: Benchmarking Google’s new TPUv2

#32

Post by Mononokay » Fri, Feb 23, 2018, 5:12 PM UTC

bitL wrote: post #21

Mononokay wrote: post #7

Earlier quoted context omitted.

There's no real need to worry about Google in the long term - nVidia can make back their money solely with their GPUs; Google probably made their expenses back this weekend with searches around the Olympics. It'd be pointless for them to not use their TPUs themselves, and their main product, Adsense, uses ML.

Are you sure about Adsense? Talked to ad pros recently and they all complained Adsense is ancient (still mySQL?) and often broken; doesn't look like Google emphasizes it despite being their cash cow, more like deep state of neglect.

Yeah, Adsense:

https://techcrunch.com/2018/02/21/google-debuts-adsense-auto...

Re: Benchmarking Google’s new TPUv2

#33

Post by dgacmu » Fri, Feb 23, 2018, 5:14 PM UTC

jrk wrote: post #25

[Edited] The top line results focus on comparing four TPUs in a rack node (which marketing cleverly named “one cloud TPU”), running ~16 bit mixed precision, to one GPU (out of 8 in a rack node), also capable of 16 bit or mixed precision, but handicapped to 32 bit IEEE 754. That is a misleading comparison. Images/$ are obviously more directly comparable, but again the emphasized comparisons are at different precision.…

That's why you scroll down the page to the cost comparison, which places it on a more even keel. They do also compare float16 on Volta. Physical packaging is irrelevant -- what matters is dollars to convergence and time to convergence.

(I'm obviously biased - I helped with parts of the cloud-side of cloud TPU - but I presume this comment stands on its own. :-)

Re: Benchmarking Google’s new TPUv2

#34

Post by boulos » Fri, Feb 23, 2018, 5:14 PM UTC

jrk wrote: post #25

[Edited] The top line results focus on comparing four TPUs in a rack node (which marketing cleverly named “one cloud TPU”), running ~16 bit mixed precision, to one GPU (out of 8 in a rack node), also capable of 16 bit or mixed precision, but handicapped to 32 bit IEEE 754. That is a misleading comparison. Images/$ are obviously more directly comparable, but again the emphasized comparisons are at different precision.…

Disclosure: I work on Google Cloud.

Depends on your metric, Jonathan! If you focus on the per dollar numbers, then it’s actually net favorable to the V100, because a second GPU over NVLINK won’t be as cost-efficient. If what you care about is raw throughput “in a single box”, then 8xV100 probably comes out ahead here.

Like someone else below though, I worry about the “hey wait a minute, changing the batch size just for the TPU seems unfair” and the whole “the LSTM didn’t converge” bit. Not a bad first draft, but hopefully the authors can do some more comparisons.

Re: Benchmarking Google’s new TPUv2

#35

Post by cobookman » Fri, Feb 23, 2018, 5:17 PM UTC

jlebar wrote: post #11

bhouston wrote: post #4

It is hard for Google to make money on these TPUs as the whole engineering cost has to be made back from its pricing on Google Cloud, where as with NVIDIA it can pay back its engineering costs via multiple mature channels (games, super computers, and multiple cloud providers.) I wonder which is higher, the cost for creating the TPUs in terms of engineering and manufacturing or the cost differential in terms of usage…

> I am surprised the TPU doesn't kick the ass of the NVIDIA chips. 30% cheaper e2e price for the company's first public offering, compared to the market leader's top-of-the-line chip sounds...pretty good to me?

30% list price. Who knows what the underlying margins are and how much cheaper Google can go with an offline agreement.

Re: Benchmarking Google’s new TPUv2

#36

Post by chapill » Fri, Feb 23, 2018, 5:17 PM UTC

KirinDave wrote: post #24

chapill wrote: post #3

I wonder if Chinese companies will use (or be allowed to use) TPUs. It seems like a pretty obvious way to have the NSA scoop up any Chinese AI advancements China may want to keep secret.

It's interesting to me that the assumption here is that government actors are interested in this as opposed to the companies hosting them.

Oh, I totally agree with you there. It's just I consider Google a government actor too.

Re: Benchmarking Google’s new TPUv2

#37

Post by boulos » Fri, Feb 23, 2018, 5:22 PM UTC

Disclosure: I work on Google Cloud.

While not perfect, I want to commend the RiseML folks for doing not only an “just out of the box” run in both regular and fp16 mode (for V100), but also adding their own LSTM experiment to the mix. We need third-party benchmarks whenever new hardware or software are being sold by vendors (reminder: I benefit from you buying Google Cloud!).

I hope the authors are able to collect some of the feedback here and update their benchmark and blog post. The question about batch size comparisons is probably the most direct, but like others, I’d encourage a run on 1, 2, 4 and 8 V100s as well.

Re: Benchmarking Google’s new TPUv2

#38

Post by gok » Fri, Feb 23, 2018, 5:24 PM UTC

The bar graph seems a little whacky. It groups the TPU (which can only do FP16) with the FP32 results from the GPUs, then puts the FP16 GPU results off to the side even though that's much closer to what the TPU is doing.

Impressive results regardless though; quite a bit faster than V100 than the paper specs would suggest.

Re: Benchmarking Google’s new TPUv2

#39

Post by deepnotderp » Fri, Feb 23, 2018, 5:24 PM UTC

olfactory wrote: post #17

chapill wrote: post #15

Earlier quoted context omitted.

Well, they do have the fastest supercomputer in the world currently and it's made with homegrown chips. No Intel ME backdoors there. Smaller chinese companies could, for a little more money, get similar performance buying 8x V100 machines from NVidia. I don't think they want to share their advancements in AI fighter pilots with USA. They have a big lead.

What is the hardest thing to accomplish with something like a TPU? Is it the IP or the fabrication? How does the TPU design offer improved performance? By leveraging IP or fabrication improvements?

Neither, it's a matrix multiply systolic array ASIC, that's been done decades ago.

There are host of Chinese companies developing similar processors.

Re: Benchmarking Google’s new TPUv2

#40

Post by gumby » Fri, Feb 23, 2018, 5:24 PM UTC

angrygoat wrote: post #2

Google claim 29x better performance-per-Watt with TPUs than contemporary GPUs[0]. Interesting to contrast that to the images-per-$ figure in this post, which is more like 2x. I assume there's a high capital cost for this new hardware, but when they scale it up I wonder if the ratio of cost TPU to GPU will trend towards the ratio of power-per-Watt between the platforms? Seems like a natural limit, even if it never qui…

> Google claim 29x better performance-per-Watt with TPUs than contemporary GPUs[0]. Interesting to contrast that to the images-per-$ figure in this post, which is more like 2x.

But you aren't paying for the electricity, you're paying for processing, which is an unconnected parameter. They only "sell" these chips per use, not on the open market

Presuming power is a major cost input (which I assume) their profit/op is much higher. So they could sell for less than an equivalent GPU and make more money. But they think they can get away with value pricing it (processing more/unit time is presumably worth it to many customers) and more power to them.

(that last bit was not an intentional pun; only noticed after typing it)