Cloud TPU Pods Break AI Training Records • Hacker News Board

Re: Cloud TPU Pods Break AI Training Records

#21

Post by zak » Sat, Jul 13, 2019, 5:42 PM UTC

rrss wrote: post #17

gok wrote: post #16

I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in…

You mean this speedup? > 1024 TPUs are twice as fast as 480 GPUs Might it be because there are twice as many?

Author of the blog post here.

We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales:

https://mlperf.org/training-results-0-6

We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

Re: Cloud TPU Pods Break AI Training Records

#22

Post by gok » Sat, Jul 13, 2019, 5:47 PM UTC

zak wrote: post #21

rrss wrote: post #17

Earlier quoted context omitted.

You mean this speedup? > 1024 TPUs are twice as fast as 480 GPUs Might it be because there are twice as many?

Author of the blog post here. We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales: https://mlperf.org/training-results-0-6 We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

What I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?

Re: Cloud TPU Pods Break AI Training Records

#23

Post by zak » Sat, Jul 13, 2019, 5:48 PM UTC

rrss wrote: post #20

boulos wrote: post #14

Earlier quoted context omitted.

Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection. I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should…

I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) wo…

Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all.

Our impression is that top-line performance comparisons independent of system size (like the comparison in the blog post) and performance-per-dollar comparisons are the easiest to understand, but we're certainly open to other ideas.

Re: Cloud TPU Pods Break AI Training Records

#24

Post by zak » Sat, Jul 13, 2019, 5:52 PM UTC

gok wrote: post #22

zak wrote: post #21

Earlier quoted context omitted.

Author of the blog post here. We submitted multiple results using various Cloud TPU v3 Pod slice sizes to show the current achievable Transformer training efficiency at several scales: https://mlperf.org/training-results-0-6 We're actively improving the whole TPU software stack, so training efficiency is likely to continue to increase over time.

What I'm really asking is: how much effective compute throughput are you able to get during Transformer training relative to the amount of theoretical raw compute available?

That's a great question. I don't have that analysis handy, but it would definitely be worth doing.

Re: Cloud TPU Pods Break AI Training Records

#25

Post by zak » Sat, Jul 13, 2019, 5:54 PM UTC

aey wrote: post #7

rrss wrote: post #6

It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…

Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.

Author of the blog post here.

I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.

Re: Cloud TPU Pods Break AI Training Records

#26

Post by zak » Sat, Jul 13, 2019, 5:55 PM UTC

ArthurBrussee wrote: post #2

Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!

Author of the blog post here.

As mentioned in other comments, I'd recommend doing a performance-per-dollar comparison in addition to looking at this pure performance comparison at maximum scale.

Re: Cloud TPU Pods Break AI Training Records

#27

Post by rrss » Sat, Jul 13, 2019, 5:58 PM UTC

zak wrote: post #23

rrss wrote: post #20

Earlier quoted context omitted.

I'm not (currently) objecting to the actual MLPerf results. I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart. There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) wo…

Thanks for the feedback. As more and more new ML system architectures emerge over the next few years, these performance comparisons will get even more complicated. Chip-to-chip comparisons and server-to-server comparisons are arbitrary, and it will sometimes be difficult to define these boundaries at all. Our impression is that top-line performance comparisons independent of system size (like the comparison in the bl…

> top-line performance comparisons independent of system size

Then do that, just don't do it all in one chart with the systems only specified in the fine print.

Re: Cloud TPU Pods Break AI Training Records

#28

Post by zak » Sat, Jul 13, 2019, 6:00 PM UTC

totoglazer wrote: post #3

ArthurBrussee wrote: post #2

Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!

I think it’s probably because the benchmark isn’t optimized for TPU Pods. Check out the BERT in 76 minutes paper for how you need to rethink the training regime to take advantage of pods.

Yes, Cloud TPU Pods are designed to train much larger models on much larger datasets. And, as you mention, if you are willing to adjust your model architectures and training algorithms to take full advantage of the hardware, you can sometimes achieve substantial gains.

Re: Cloud TPU Pods Break AI Training Records

#29

Post by rrss » Sat, Jul 13, 2019, 6:05 PM UTC

zak wrote: post #19

rrss wrote: post #6

It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…

Author of the blog post here. Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story. The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comp…

Why performance per dollar over performance per watt?

Performance per watt doesn't care where the systems are (on-prem vs cloud) and should track perf/$ (unless perf/$ is mostly determined by subsidies).

Re: Cloud TPU Pods Break AI Training Records

#30

Post by aey » Sat, Jul 13, 2019, 6:17 PM UTC

zak wrote: post #25

aey wrote: post #7

Earlier quoted context omitted.

Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.

Author of the blog post here. I'd recommend doing a performance-per-dollar comparison before drawing this conclusion.

What’s the perf per watt difference?

It’s hard to compare perf per dollar. Your electricity costs and GPU costs maybe vastly different from mine.