Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Cloud TPUs in Beta

cloudplatform.googleblog.com

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Cloud TPUs in Beta

#121

Post by twtw »

jlebar wrote: post #116
deepnotderp wrote: post #71

Earlier quoted context omitted.

This makes no sense, the V100 has more memory bandwidth than both the TPU and TPUv2

V100 has 900gb/s memory bandwidth [0]. TPUv2 has 600gb/s per chip x 4 chips, so 2400gb/s [1]. As we've discussed elsewhere [2], comparing TPUv2 to V100 on a per chip basis doesn't make much sense. Who cares how many chips are on the board? If Google announced tomorrow that TPUv3 is coming out, which is identical to TPUv2 but the four chips are glued together, nobody would care. The questions that we should instead be…

I responded to your other comment to disagree, and I'll do so again here.

Nobody is comparing DGX1-V to a single TPUv2 chip, because it doesn't make any sense to do so. they are totally different kinds of machines. But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip.

It only makes sense to compare 4xTPUv2 to 1xV100 if they are equivalent in some meaningful metric, like total die size, power, etc.

In lieu of any available data, I'm going to continue to assume that each TPUv2 chip is roughly comparable in terms of power & die size to each V100 chip. If this was grossly wrong, I would expect that all four would be condensed into a single chip, which would dramatically increase the performance of the interconnects.

We could resolve this rapidly if there were any data available about die size, TDP, anything of TPUv2.

Re: Cloud TPUs in Beta

#122

Post by Cyph0n »

boulos wrote: post #88
Cyph0n wrote: post #20

Earlier quoted context omitted.

A better comparison would be the f1.16xlarge[1] instance @ ~$4/hr. It comes with 8 FPGAs (12 Gbps link) and 64 vCPUs. [1]: https://aws.amazon.com/ec2/instance-types/f1/ Edit: I'm genuinely curious about why this comment is getting downvotes.

Disclosure: I work on Google Cloud. I didn’t downvote you, but presumably people disagree with “Here’s an FPGA” as comparable to being given a working piece of hardware. That is, would you have said that the best comparison to a V100 is this same FPGA box? I (and others) get what you were trying to say: TPUs are ASICs that aren’t general purpose at all, so an FPGA is a better comparison than a more general purpose GP…

Fair enough. I wonder if there are any model compilers targeting FPGA training backends...

Re: Cloud TPUs in Beta

#123

Post by jlebar »

twtw wrote: post #121
jlebar wrote: post #116

Earlier quoted context omitted.

V100 has 900gb/s memory bandwidth [0]. TPUv2 has 600gb/s per chip x 4 chips, so 2400gb/s [1]. As we've discussed elsewhere [2], comparing TPUv2 to V100 on a per chip basis doesn't make much sense. Who cares how many chips are on the board? If Google announced tomorrow that TPUv3 is coming out, which is identical to TPUv2 but the four chips are glued together, nobody would care. The questions that we should instead be…

I responded to your other comment to disagree, and I'll do so again here. Nobody is comparing DGX1-V to a single TPUv2 chip, because it doesn't make any sense to do so. they are totally different kinds of machines. But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip. It only makes sense to compare 4xTPUv2 to 1xV100 if they are equivalent in some meaningful metric, like total die…

> But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip.

I agree that some people are doing that. Marketing, I suppose. But that comparison is explicitly not the point of my parent post. I'm comparing the "shapes" of the chips -- specifically, the compute/memory and compute/memory-bandwidth ratios. These ratios stay the same regardless of whether you multiply the chips by 4 or by 400.

The point I was trying to make is that V100 has a higher peak-compute-to-memory(-bandwidth) ratio than TPUv2. This much seems clear from the arithmetic. Whether this matters in practice, I don't know, but I think it is relevant if one believes (as I do, based on the evidence I have as an author of an ML compiler targeting the V100) that the V100 is starved for memory bandwidth.

> In lieu of any available data, I'm going to continue to assume that each TPUv2 chip is roughly comparable in terms of power & die size to each V100 chip. If this was grossly wrong, I would expect that all four would be condensed into a single chip, which would dramatically increase the performance of the interconnects.

I'm sure Google's hardware engineers operate under a lot of constraints that I'm not aware of; I'm not about to make assumptions. But more to the point, as we've said, things like die size and TDP don't directly affect consumers. The questions we have to ask are, how fast can you train your model, and at what cost?

Just as you don't like it when people (incorrectly, I agree) insist on comparing one V100 to four TPUs, because that's totally arbitrary (why not compare one V100 to 128 TPUs?), I don't like it when people insist on comparing TPUv2 to V100 on arbitrary metrics like die size, or peak flops/chip, or whatever. So I disagree that we could resolve anything if we had more info about the TPUv2 chip itself. None of that matters.

Re: Cloud TPUs in Beta

#124

Post by neilmovva »

jlebar wrote: post #61
twtw wrote: post #26

Some things: A "single TPU" is 4 ASICs. It is not clear if it makes sense to compare a "single TPU" to a "single GPU." As a point of reference, NVIDIA's numbers are 6 hours for Resnet-50 on Imagenet when training with 8xV100. From a naive extrapolation, 4xV100 would probably take ~12 hours and 1xV100 about two days. Google has previously only compared TPUs to K80, so it will be interesting to see some benchmarks that…

> A "single TPU" is 4 ASICs. It is not clear if it makes sense to compare a "single TPU" to a "single GPU." Why does the number of chips matter? Put another way, suppose Google tomorrow announced Cloud TPU v3 which was one ASIC identical in all ways to four v2 ASICs glued together. Would that be notable in any way? Seems like it would be a nop to me. I think what matters is, how fast can you train a model, and at wha…

It matters when defining parallel work distribution. Unless memory bandwidth is homogeneous across the whole board (i.e. each TPU on a board gets 600 GB/s to its peers), we can't do model parallelism across ASICs efficiently, and must fall back to data parallelism. Which is fine, until you run into limits on maximum batchsize (e.g. up to 8192, as FAIR was able to manage [1] with some tweaks to SGD).

[1] https://arxiv.org/abs/1706.02677

Re: Cloud TPUs in Beta

#125

Post by twtw »

jlebar wrote: post #123
twtw wrote: post #121

Earlier quoted context omitted.

I responded to your other comment to disagree, and I'll do so again here. Nobody is comparing DGX1-V to a single TPUv2 chip, because it doesn't make any sense to do so. they are totally different kinds of machines. But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip. It only makes sense to compare 4xTPUv2 to 1xV100 if they are equivalent in some meaningful metric, like total die…

> But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip. I agree that some people are doing that. Marketing, I suppose. But that comparison is explicitly not the point of my parent post. I'm comparing the "shapes" of the chips -- specifically, the compute/memory and compute/memory-bandwidth ratios. These ratios stay the same regardless of whether you multiply the chips by 4 or by…

Alright, I understand better now what you are saying. I'm eager to see some benchmarks that can answer those meaningful questions.

Thank you for your courteous reply.

Re: Cloud TPUs in Beta

#126

Post by oneshot908 »

boulos wrote: post #120
londons_explore wrote: post #118

Earlier quoted context omitted.

It is documented here: https://github.com/tensorflow/tensorflow/blob/master/tensorf...

That doesn’t prove that the chip operates at 16 bits. For example, we could do 18-bit multipliers (or anything >= 16) and still use 16-bit floats.

ATI demonstrated FP24 was frickin' awesome over a decade and a half ago. it wouldn't surprise me in the least if you went somewhere like that, but it perplexes me as to why you think that's secret sauce in any way long after ATI nearly destroyed NVIDIA with FP24 back in the early days of DirectX 9 and NV3x.

Re: Cloud TPUs in Beta

#127

Post by socceroos »

londons_explore wrote: post #117
sabalaba wrote: post #106

Earlier quoted context omitted.

Any plans to support AMD GPUs and the Radeon Open Compute project? The AI/ML community really needs viable alternatives to NVIDIA, otherwise they will continue to flex pricing power. Google, via TensorFlow, is in a phenomenal position to promote open source alternatives to the proprietary Deep Learning software ecosystem that we see today with CUDA/CuDNN.

Google would happily accept patches to enable support for it. AMD hopefully has a team writing such patches now. It makes business sense for them to do so. Google is getting even more price gouging from Nvidia than the general public, and has even more incentive to level the playing field.

Or the opposite - they're getting nice savings in return for not actively developing or encouraging CUDA/cuDNN alternatives.

Re: Cloud TPUs in Beta

#128

Post by deepnotderp »

jlebar wrote: post #123
twtw wrote: post #121

Earlier quoted context omitted.

I responded to your other comment to disagree, and I'll do so again here. Nobody is comparing DGX1-V to a single TPUv2 chip, because it doesn't make any sense to do so. they are totally different kinds of machines. But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip. It only makes sense to compare 4xTPUv2 to 1xV100 if they are equivalent in some meaningful metric, like total die…

> But for some reason everyone is comparing a cluster of 4 TPUv2 chips to a single V100 chip. I agree that some people are doing that. Marketing, I suppose. But that comparison is explicitly not the point of my parent post. I'm comparing the "shapes" of the chips -- specifically, the compute/memory and compute/memory-bandwidth ratios. These ratios stay the same regardless of whether you multiply the chips by 4 or by…

Well, if you ignore power consumptiom because ",it doesn't matter to the end user", you're talking about economic comparisons, not technical comparisons.

BTW, I absolutely agree that memory bandwidth is the bottleneck, I've built my company around that assertion and the data for that exists (Mitra's publications come to mind)

Re: Cloud TPUs in Beta

#129

Post by londons_explore »

oneshot908 wrote: post #126
boulos wrote: post #120

Earlier quoted context omitted.

That doesn’t prove that the chip operates at 16 bits. For example, we could do 18-bit multipliers (or anything >= 16) and still use 16-bit floats.

ATI demonstrated FP24 was frickin' awesome over a decade and a half ago. it wouldn't surprise me in the least if you went somewhere like that, but it perplexes me as to why you think that's secret sauce in any way long after ATI nearly destroyed NVIDIA with FP24 back in the early days of DirectX 9 and NV3x.

I don't think it is secret sauce. If you're gonna let customers send operations to these TPU's, one could figure out what kind of multiplier is used almost immediately upon inspection of a few inputs and outputs.

Re: Cloud TPUs in Beta

#130

Post by tveita »

Is this just go-faster-juice for Tensorflow code or does it have other implications? If you train on TPUs can you still run the model efficiently elsewhere?
Post reply on HN