Live data from Hacker News

Benchmarking Google’s new TPUv2

blog.riseml.com

71–80 of 95 posts

Re: Benchmarking Google’s new TPUv2

#71

Earlier quoted context omitted.

Volta V100 already has "tensor cores" which are basically little matrix multiplication ASICs.

That's what I said. The microarchctiecture has many unnecessary things and it's not optimized as a whole for deep learning.

I believe it was either the last MICRO* or the one before that when Dally addressed this point. The specialized hardware for graphics ends up comprising such a small portion of the overall chip that it wasn't worth it to remove it. The "GPUs were made for graphics thus aren't good for DL" argument really doesn't hold a lot of water IMO.

* It might've been a difference conference now that I think of it.

Re: Benchmarking Google’s new TPUv2

#72

Wait but, the batch size is 8x bigger for the TPU? That's not a fair comparison; increasing batch size always speeds things up...

Author here.

Note that the TPU supports larger batch sizes because it has more RAM. We tested multiple batch sizes for GPUs and reported the fastest one. We'll try increasing the batch sizes as far as possible and report. The overall comparison will likely not change by much - we saw speed increases of around 5% doubling the batch size from 64 to 128. (https://www.tensorflow.org/performance/benchmarks also reports numbers for batch sizes of 32 and 64 on the P100)

Re: Benchmarking Google’s new TPUv2

#74

Does this take into account the fact that you might need fewer epochs if you reduce the batch size? (as is done for the CPU?)

Author here.

No, this is really only comparing the throughput on the devices. A thorough comparison should really focus on time to reach a certain quality - including all of the tricks available for a certain architecture.

Re: Benchmarking Google’s new TPUv2

#76
post #33
post #25

[Edited] The top line results focus on comparing four TPUs in a rack node (which marketing cleverly named “one cloud TPU”), running ~16 bit mixed precision, to one GPU (out of 8 in a rack node), also capable of 16 bit or mixed precision, but handicapped to 32 bit IEEE 754. That is a misleading comparison. Images/$ are obviously more directly comparable, but again the emphasized comparisons are at different precision.…

That's why you scroll down the page to the cost comparison, which places it on a more even keel. They do also compare float16 on Volta. Physical packaging is irrelevant -- what matters is dollars to convergence and time to convergence. (I'm obviously biased - I helped with parts of the cloud-side of cloud TPU - but I presume this comment stands on its own. :-)

To be clear, I had read the whole post, I was just being terse since the emphasis seemed to be so heavily on an apples to bananas comparison (I believe 100% of the results cited in the prose, many in bold, are with mismatched precision and batch size), with minimal articulation of the many axes of nuance here. Precision isn't defined at all in the LSTM case, and could easily be the cause of the failure of the TPU run to converge where the GPU runs do. To a non-expert audience I think the end result is confusing and misleading.

Also, while I certainly agree that the performance/dollar comparison is highly relevant to customers at a given instant, that may only tell us that Google is subsidizing this hardware now that they've deployed it, and/or that, lacking serious competition, NVIDIA has been building crazy margins into their P100/V100 prices. In understanding fundamental technological tradeoffs, and even the limits of what the pricing in a more competitive market could be, it is relevant to compare performance per unit of hardware resources (mm^2, die/package, watt, GB of HBM, etc.)

In short, these comparisons are hard, and there is no one which tells a complete story. I pushed back because, while the post includes some nuance, it brushes a great deal under the rug and focuses primarily on problematic comparison

(Further disclosure: I'm at least the third person in this sub-thread with some Google Brain/Cloud affiliation. I am speaking in my independent academic voice. I also think TPUs are great, having them publicly available now is great, and competition and diversity of architectural approach in accelerators is great. I appreciate the effort of the authors, but think the subtlety of these comparisons requires serious care.)

Re: Benchmarking Google’s new TPUv2

#77

Wait but, the batch size is 8x bigger for the TPU? That's not a fair comparison; increasing batch size always speeds things up...

Author here. Note that the TPU supports larger batch sizes because it has more RAM. We tested multiple batch sizes for GPUs and reported the fastest one. We'll try increasing the batch sizes as far as possible and report. The overall comparison will likely not change by much - we saw speed increases of around 5% doubling the batch size from 64 to 128. ( https://www.tensorflow.org/performance/benchmarks also reports n…

Disclosure: I work on Google Cloud.

Oh! You should definitely say that. It's semi-reasonable then to choose the batch size that is optimal for the part. It'd be good to make sure this isn't why your LSTM didn't converge though...

Re: Benchmarking Google’s new TPUv2

#78
post #34
post #25

[Edited] The top line results focus on comparing four TPUs in a rack node (which marketing cleverly named “one cloud TPU”), running ~16 bit mixed precision, to one GPU (out of 8 in a rack node), also capable of 16 bit or mixed precision, but handicapped to 32 bit IEEE 754. That is a misleading comparison. Images/$ are obviously more directly comparable, but again the emphasized comparisons are at different precision.…

Disclosure: I work on Google Cloud. Depends on your metric, Jonathan! If you focus on the per dollar numbers, then it’s actually net favorable to the V100, because a second GPU over NVLINK won’t be as cost-efficient. If what you care about is raw throughput “in a single box”, then 8xV100 probably comes out ahead here. Like someone else below though, I worry about the “hey wait a minute, changing the batch size just f…

Author here.

Thanks for your feedback. As I noted above, we will report further results with larger batch sizes (and smaller ones for the TPU). The LSTM not converging is one of the experiences we wanted to share. We are working on solving this issue and will update the post accordingly. Our goal is really a fair and valuable comparison, which is not easy, so we value all of the feedback.

Re: Benchmarking Google’s new TPUv2

#79
post #77

Earlier quoted context omitted.

Author here. Note that the TPU supports larger batch sizes because it has more RAM. We tested multiple batch sizes for GPUs and reported the fastest one. We'll try increasing the batch sizes as far as possible and report. The overall comparison will likely not change by much - we saw speed increases of around 5% doubling the batch size from 64 to 128. ( https://www.tensorflow.org/performance/benchmarks also reports n…

Disclosure: I work on Google Cloud. Oh! You should definitely say that. It's semi-reasonable then to choose the batch size that is optimal for the part. It'd be good to make sure this isn't why your LSTM didn't converge though...

I tested many different batch sizes for the LSTM, so I am pretty confident it's not the reason.

Re: Benchmarking Google’s new TPUv2

#80

Earlier quoted context omitted.

I would not see things stagnating but it seems like there's a potential for the individual to get cut out of this excitement if each of these entities is keeping it's chips close to it's chest. The era of the mainframe, with each provider competing with a custom chip, wasn't necessarily beneficial for individual buying computer power.

Industries go through cycles of innovation and concentration. During innovation cycles, many new non-standard products appear with innovative solutions, the entire pie grows really fast. As growth eventually stabilizes, standards become more relevant and consolidation happens, eventually leading to a stagnation that makes the industry ripe for disruption and change again. If you look at processors, you see that with…

I second this.

With cloud computing, we are essentially going back to the era of time-shared mainframes with remote access.

Post reply on HN