Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

blog.riseml.com

Post reply on HN
Print view View original

81–90 of 132 posts

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#81

Post by eanzenberg »

ndesaulniers wrote: post #76
eanzenberg wrote: post #74

Earlier quoted context omitted.

AWS yes. Nvidia, not so sure. When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card.

> When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card. Those cards are also irrelevant to the comparison as they can't be bought in large capacities for ML workloads. We're talking about Titan-V's and DGX-1's here.

Are you suggesting the Titan-V price is subsidized by Nvidia?

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#82

Post by ndesaulniers »

elmarhaussmann wrote: post #66
kyloon wrote: post #58

Excellent work. Do you have plans to open source the scripts/implementation details used to reproduce the results? Would be great if others can also validate and repeat the experiment for future software updates (e.g. TensorFlow 1.8) as I expect there will be some performance gain for both TPU and GPU by CUDA and TensorFlow optimizations. Sidenote: Love the illustrations that accompany most of your blog posts, are th…

Happy you like the post! The implementations we used are open source (we reference the specific revisions), so reproducing results is possible right now. We haven't thought about publishing our small scripts around that (there's not much to it), but it's a good idea. There's also work towards benchmarking suites like DAWNBench ( https://dawn.cs.stanford.edu/benchmark/ ). The illustrations are from an artist/designer…

> The illustrations are from an artist/designer we contract from time to time. I agree, his work is awesome!

Kudos to them; they are awesome!

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#83

Post by smallnamespace »

jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

> About 1/2 the cost for similar performance.

I would expect a dedicated accelerator to need at least a 5-10X advantage to outweigh all the other infrastructure and ecosystem costs.

GPUs are more useful for a wide variety of data-parallel tasks, and many more NN frameworks work on top of CUDA than work on the TPU.

In terms of horizontal scalability, nvidia has been rapidly iterating on increasing both memory and interlink bandwidth (including NVSwitch [1]), while each 'TPU' is actually 4 chips interconnected so likely has less upward scalability.

Also note that the tensor cores on a V100 take roughly 25-30% of the actual area. If Nvidia wanted to, they could probably easily make a pure tensor chip that beat the TPU in performance, could be produced in volume on their existing process, and also had full compatibility with their entire stack.

All in all, a 2x price/performance advantage for a hyper-specialized accelerator is basically a loss, just like how nobody installs a Soundblaster card anymore, how consumer desktops don't run discrete GPUs even though integrated graphics are a few times slower, or

[1] https://www.nextplatform.com/2018/04/04/inside-nvidias-nvswi...

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#84

Post by dgacmu »

verall wrote: post #73
dgacmu wrote: post #31

Earlier quoted context omitted.

Yes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lac…

The new "datacenter" restriction only applies to GeForce branded cards. The Titan V is now called the "NVIDIA Titan V" and with no GeForce branding to be found anywhere. So the restriction applies to the 1080ti but _not_ the titan V. I completely agree the restriction is total bullshit but it's important to get the facts straight.

Not according to the statement from NVidia quoted in this article: https://www.cnbc.com/2017/12/27/nvidia-limits-data-center-us...

It applies to both GeForce and Titan.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#85

Post by ndesaulniers »

eanzenberg wrote: post #81
ndesaulniers wrote: post #76

Earlier quoted context omitted.

> When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card. Those cards are also irrelevant to the comparison as they can't be bought in large capacities for ML workloads. We're talking about Titan-V's and DGX-1's here.

Are you suggesting the Titan-V price is subsidized by Nvidia?

Let's revisit your original point:

> Pricing is kind of irrelevant since they can subsidize this to create that story.

You seemed to imply they == google. My point is that it could cut both ways.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#86

Post by puzzle »

deepnotderp wrote: post #38
jacksmith21006 wrote: post #11

Earlier quoted context omitted.

The chart has 6.7 per hour for 3186 images Google and 12.2 per hour for 3128 AWS. Or maybe reading it wrong? That is close to half has much to use Google is it not? BTW, The TPUs are also about twice as fast also. Sounds like Google is pretty far ahead of Nvidia. Which really just makes sense as Google does the entire stack and just going to have the data to optimize the silicon. About half the cost is hype? I want i…

fwiw, the "TPU instance " has more than one tpu chip on it.

It has more than the chips, too, since the TPUs can't run a TCP/IP stack, gRPC server, etc.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#87

Post by rprenger »

jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

Full disclosure, I currently work at Nvidia on speech synthesis.

You can definitely do this on a GPU. We use the older auto-regressive WaveNets (not Parallel Wavenet) for inference on GPUs, with the newly released nv-wavenet code. Here's a link to a blog post about it:

https://devblogs.nvidia.com/nv-wavenet-gpu-speech-synthesis

That code will generate audio samples at 48khz, or if you're worried about throughput, it'll do a batch of 320 parallel utterances at 16khz.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#88

Post by dimitry12 »

An important hidden cost here is coding a model which can take advantage of mixed-precision training. It is not trivial: you have to empirically discover scaling factors for loss functions, at the very least.

It's great that there is now wider choice of (pre-trained?) models formulated for mixed-precision training.

When I was comparing Titan V (~V100) and 1080ti 5 months ago, I was only able to get 90% increase in forward-pass speed for Titan V (same batch-size), even with mixed-precision. And that was for an attention-heavy model, where I expected Titan V to show its best. Admittedly, I was able to use almost double the batch-size on Titan V, when doing mixed-precision. And Titan V draws half the power of 1080ti too :)

At the end my conclusion was: I am not a researcher, I am a practitioner - I want to do transfer learning or just use existing pre-trained models - without tweaking them. For that, tensor cores give no benefit.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#89

Post by bitL »

dimitry12 wrote: post #88

An important hidden cost here is coding a model which can take advantage of mixed-precision training. It is not trivial: you have to empirically discover scaling factors for loss functions, at the very least. It's great that there is now wider choice of (pre-trained?) models formulated for mixed-precision training. When I was comparing Titan V (~V100) and 1080ti 5 months ago, I was only able to get 90% increase in fo…

How did you get your hands on Titan V 5 months ago? I still can't find it anywhere in retail in EU...

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#90

Post by shaklee3 »

ndesaulniers wrote: post #65
shaklee3 wrote: post #45

Earlier quoted context omitted.

I'm not sure what you mean by google does the entire stack. Nvidia writes all of the major CUDA libraries used behind the scenes in the NN libraries, such as cuDNN, cuBLAS, etc. Nvidia can likely improve their hardware significantly faster/more efficiently than Google can because their entire business depends on it. Google has incentive for improving their TPU for internal use, but they don't make any money by sellin…

> I'm not sure what you mean by google does the entire stack. Consider that Google has some of the best machine learning researchers, compiler engineers, hardware engineers, and infrastructure in the business working on this.

Huh? Machine learning and infrastructure Engineers, yes. Compiler and Hardware engineers? No. What gives you reason to believe they have a lead in either of those departments other than they have a lot of money? They're forced to use the same foundry as Nvidia, and their Hardware team is likely significantly smaller.
Post reply on HN