Cloud TPU Pods Break AI Training Records
cloud.google.com
Cloud TPU Pods Break AI Training Records
1–10 of 48 posts
Re: Cloud TPU Pods Break AI Training Records
#2Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!
Re: Cloud TPU Pods Break AI Training Records
#3Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!
I think it’s probably because the benchmark isn’t optimized for TPU Pods. Check out the BERT in 76 minutes paper for how you need to rethink the training regime to take advantage of pods.
Re: Cloud TPU Pods Break AI Training Records
#4What's in the picture? Can't figure out the scale. Are those like server racks or breadboards or ...?
Re: Cloud TPU Pods Break AI Training Records
#5What's in the picture? Can't figure out the scale. Are those like server racks or breadboards or ...?
Server racks. The colored things are all cables.
Re: Cloud TPU Pods Break AI Training Records
#6It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark).
Seriously.
Transformer: 1024 TPUs are twice as fast as 480 GPUs.
Resnet50: 1536 GPUs are about as fast as 1024 TPUs.
SSD: 1024 TPUs are twice as fast as 240 GPUs.
Great.
Re: Cloud TPU Pods Break AI Training Records
#7It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
Re: Cloud TPU Pods Break AI Training Records
#8It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
You might want to check the DAWN benchmark, it does just that:
[1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...
Re: Cloud TPU Pods Break AI Training Records
#9It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
Re: Cloud TPU Pods Break AI Training Records
#10It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
You might want to check the DAWN benchmark, it does just that: [1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...
Thanks, I've seen dawnbench, and it's definitely better than this presentation.
Unfortunately, their cost numbers aren't actual cost, as they don't allow using AWS spot instance pricing (which is what fast.ai does).