What's in the picture? Can't figure out the scale. Are those like server racks or breadboards or ...?
Here's a higher resolution version: https://storage.googleapis.com/gweb-cloudblog-publish/origin...
Cloud TPU Pods Break AI Training Records
11–20 of 48 posts
Re: Cloud TPU Pods Break AI Training Records
#12Earlier quoted context omitted.
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
[deleted]
Re: Cloud TPU Pods Break AI Training Records
#13Earlier quoted context omitted.
Results are disappointing. 2x speed up over general purpose GPUs doesn’t justify a whole new hardware architecture.
When Moores law is finally dead and buried (e.g. 5nm), new architectures will be all that's left. Seems like a great time to start down the new architecture path.
Have semiconductor companies not been on the new architecture path already? It's not like they've just been doing die shrinks this whole time.
Re: Cloud TPU Pods Break AI Training Records
#14It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Disclosure: I work on Google Cloud.
I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection.
I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should I do inference on this chip from ? I dunno, what’s it going to cost?”). Having to keep going back to some reference box is undesirable, but there’s nothing stopping someone from making a spreadsheet equivalent that translates these to dollars on the cloud providers (harder for on-prem submissions which reopens the “rent vs buy” debate).
Re: Cloud TPU Pods Break AI Training Records
#15Earlier quoted context omitted.
You might want to check the DAWN benchmark, it does just that: [1] https://dawn.cs.stanford.edu/benchmark/index.html#imagenet-t...
Thanks, I've seen dawnbench, and it's definitely better than this presentation. Unfortunately, their cost numbers aren't actual cost, as they don't allow using AWS spot instance pricing (which is what fast.ai does). https://www.fast.ai/2018/04/30/dawnbench-fastai/
It was decided to disallow spot pricing and preemptible VMs because it’s super variable. On the day of a CVPR submission or something, you’re not going to pay the lowest rate (spot) nor perhaps be likely to get the VMs anyway (both spot and preemptible).
Said another way, spot pricing results aren’t as reproducible in the research sense. It’s also a similar constant factor between the main providers. We felt that once you knew relative price/perf, you can choose to do whatever economic analysis you’d prefer (e.g., maybe if you don’t have any datacenter space yourself, there is no price you’d pay for hardware on-premises, or maybe you are willing to use spot or preemptible).
Re: Cloud TPU Pods Break AI Training Records
#16I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in the TPUs.
Re: Cloud TPU Pods Break AI Training Records
#17I'm curious how (absolutely) efficient the Transformer training is even on the TPUs. The results from self-attention models are really impressive but unfortunately their topology makes them very difficult to implement efficiently in silicon. It often becomes purely a question of memory bandwidth, because you're not doing much math per weight on each iteration. I wonder if the speedup is from the use of on-chip HBM in…
You mean this speedup?
> 1024 TPUs are twice as fast as 480 GPUs
Might it be because there are twice as many?
Re: Cloud TPU Pods Break AI Training Records
#18Is it me or are those results somewhat underwhelming if anything? Dedicated hardware for a 2x speedup at best, tossup for most results, and only competes in some categories. Not to be just a NVIDIA fan here, surely there is value in dedicated training hardware, but just surprising that benefit isn't bigger!
Disclosure: I work on Google Cloud (even with Zak sometimes).
The DGX-2h is a beast! Don’t parse this as “huh, TPU Pods are about the same as just a few V100s”. The data sheet [1] is probably the easiest to follow, but their writeup is more informative [2].
These are souped up V100s, with awesome networking, which is pretty similar in style to a TPU Pod. So I’d say that they’re both purpose built systems for distributed ML training. The name for the NVIDIA system is even “DGX SuperPOD” :).
[1] https://www.nvidia.com/content/dam/en-zz/es_em/Solutions/Dat...
[2] https://devblogs.nvidia.com/dgx-superpod-world-record-superc...
Re: Cloud TPU Pods Break AI Training Records
#19It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Author of the blog post here.
Cloud TPUs are designed to maximize performance-per-dollar, so you are right that pure performance comparisons at maximum scale don't tell the whole story.
The most straightforward performance-per-dollar comparisons would be among several different hardware configurations across the major public clouds. However, we haven't yet seen any other public cloud MLPerf submissions at scales comparable to Cloud TPU Pods, so there isn't currently a strong baseline available for comparison. It's also not clear whether public cloud networking will ultimately be able to match the performance of the network hardware that was used to produce the largest-scale on-premise MLPerf submissions.
Re: Cloud TPU Pods Break AI Training Records
#20It's disappointing there is no attempt at normalizing results. It's not perf/$, or perf/W, or perf/chip, or anything that might be useful - it seems to just be perf/(the largest machine google/nvidia could afford to put together for a given benchmark). Seriously. Transformer: 1024 TPUs are twice as fast as 480 GPUs. Resnet50: 1536 GPUs are about as fast as 1024 TPUs. SSD: 1024 TPUs are twice as fast as 240 GPUs. Grea…
Disclosure: I work on Google Cloud. I know. Within MLperf, to reach consensus, it was agreed that “dollars are great, but also game-able via price cuts, lets normalize to a reference system”. It’s not the largest box though, it’s one that everyone was able to provide, so that it lets you compare through one level of indirection. I don’t agree with it, because it makes it so hard to compare to funky hardware (“Should…
I'm not (currently) objecting to the actual MLPerf results.
I'm objecting to the fact that your marketing team (at least, I hope it was your marketing team) chose to represent 5 different GPU systems as the same color and 3 different TPU systems as another color in one bar chart.
There's a book with a chapter or two on how to reasonably compare the performance of computer systems. Given that the author (Patterson) works on TPUs, I would expect better.