Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

blog.riseml.com

Post reply on HN
Print view View original

71–80 of 132 posts

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#71

Post by PaulHoule »

eanzenberg wrote: post #50
jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

The impression I got was opposite: TPU is not the hot shit that Google claims it is. Pricing is kind of irrelevant since they can subsidize this to create that story.

I know an engineer who prototypes GPU-like systems with FPGA and he has told me to be skeptical about performance miracles.

No matter how fast a system is on the inside you have to get data in and out of it -- at the very least to memory. SRAM takes too much area and there is a limit DRAM bandwidth despite technologies such as eDRAM and HBM. Some tasks are compute intensive, but for general tasks, a processor that is 100x faster would need 100x faster memory to really be 100x faster.

Thus advances in real-life performance are likely to be more like a factor of 2.

For training I never pay full price in the AWS cloud, rather I run interruptable instances and pay a fraction of the list price. People I know who train in the Google cloud seem to get interrupted all the time even though they are paying full price.

Inference is another story. Once you have the trained model, you will usually need to run inference many many more times than you run training and this gets more so the bigger scale you are running at. That hits your unit costs and it is where you need to pinch every penny.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#72

Post by [deleted] »

nojvek wrote: post #9
jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

Half the cost? Where are you reading that? Yeah on demand rental in AWS is expensive, but both long term and buying V100 yourself is significantly cheaper. Cloud companies have pretty fat margins on on demand rentals. You can’t buy a TPU, it’s a cloud only thing. They also show it’s not a huge difference in both perf and time to converge (albeit only one architecture) I would say kudos to V100 and this benchmark that…

[deleted]

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#73

Post by verall »

dgacmu wrote: post #31
samfisher83 wrote: post #23

>For GPUs, there are further interesting options to consider next to buying. For example, Cirrascale offers monthly rentals of a server with four V100 GPUs for around $7.5k (~$10.3 per hour). However, further benchmarks are required to allow a direct comparison since the hardware differs from that on AWS (type of CPU, memory, NVLink support etc.). Can't you just buy some 1080s for cheaper than this. I understand ther…

Yes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lac…

The new "datacenter" restriction only applies to GeForce branded cards. The Titan V is now called the "NVIDIA Titan V" and with no GeForce branding to be found anywhere.

So the restriction applies to the 1080ti but _not_ the titan V. I completely agree the restriction is total bullshit but it's important to get the facts straight.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#74

Post by eanzenberg »

ndesaulniers wrote: post #67
eanzenberg wrote: post #64

Earlier quoted context omitted.

At the pricing extreme, Google could make their TPUs free to use and charge elsewhere in their cloud. This shows that literal pricing is pretty irrelevant.

So could AWS/Nvidia.

AWS yes. Nvidia, not so sure. When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#75

Post by scottlegrand2 »

What they're not saying is that one can't use all nvlink bandwidth for gradient reduction on a DGX-1V with only 4 GPUs because nvlink is composed of 2 8-node rings. And given the data parallel nature of this benchmark, I'm very interested in where time was spent on each architecture.

That said, they fixed this on NVSwitch so it's just another HW hiccup like int8 was on Pascal.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#76

Post by ndesaulniers »

eanzenberg wrote: post #74
ndesaulniers wrote: post #67

Earlier quoted context omitted.

So could AWS/Nvidia.

AWS yes. Nvidia, not so sure. When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card.

> When you buy a 1080ti you are competing with gamers and miners (and maybe others). There's nothing to subsidize, in fact those cards are selling above MSRP, because they aren't selling an ecosystem but a physical card.

Those cards are also irrelevant to the comparison as they can't be bought in large capacities for ML workloads. We're talking about Titan-V's and DGX-1's here.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#77

Post by ndesaulniers »

baybal2 wrote: post #42
dgacmu wrote: post #31

Earlier quoted context omitted.

Yes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lac…

Hire people to buy 1080 in retail. This problem is solvable easily.

It's hard to find 1080[ti]+ in retail. Whenever they become available they sell out pretty quickly.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#78

Post by ndesaulniers »

deepnotderp wrote: post #38
jacksmith21006 wrote: post #11

Earlier quoted context omitted.

The chart has 6.7 per hour for 3186 images Google and 12.2 per hour for 3128 AWS. Or maybe reading it wrong? That is close to half has much to use Google is it not? BTW, The TPUs are also about twice as fast also. Sounds like Google is pretty far ahead of Nvidia. Which really just makes sense as Google does the entire stack and just going to have the data to optimize the silicon. About half the cost is hype? I want i…

fwiw, the "TPU instance " has more than one tpu chip on it.

The architectures are so radically different that I don't think it makes sense to try to compare anything but the whole system performance. Trying to do a 1 to 1 comparison for a core or a chip becomes pretty nebulous because the architectures are radically different.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#79

Post by dgacmu »

baybal2 wrote: post #42
dgacmu wrote: post #31

Earlier quoted context omitted.

Yes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lac…

Hire people to buy 1080 in retail. This problem is solvable easily.

It's not about getting the cards (though supplies are limited because of cryptocurrency mining, but you could buy Titan V's off the shelf in batches of 2). It's about whether or not you're big enough of a target for Nvidia's lawyers if you violate the agreement and actually build a datacenter out with them.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#80

Post by ndesaulniers »

eanzenberg wrote: post #50
jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

The impression I got was opposite: TPU is not the hot shit that Google claims it is. Pricing is kind of irrelevant since they can subsidize this to create that story.

Did you get that impression from this line in the article?

> While the V100s perform similarly fast, the higher price and slower convergence of the implementation results in a considerably higher cost-to-solution.

Post reply on HN