Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50 • Hacker News Board

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#41

Post by shaklee3 » Thu, Apr 26, 2018, 3:21 PM UTC

jacksmith21006 wrote: post #28

shaklee3 wrote: post #24

Earlier quoted context omitted.

Maybe, maybe not. They have the advantage that they make the hardware, so they're not paying as much retail as nvidia is charging them for their cards. I don't think there's any way you can say the TPU is cheaper compared to buying your own system. If Google decides to release it to the public, that's a different story. Also, keep in mind that Google allows you to mix and match the CPU core count to GPU, whereas AWS…

That is true. But the cost of running is where all the cost is at really not so much in making the chips. Yes I can say it is a lot cheaper. That is what this article is all about. You can do about twice the images per dollar using the TPUs with GCP versus using Nvidia with AWS. Or what am I missing? BTW, Google has released to the general public. What are you talking about? "Google’s AI chips are now open for public…

You misunderstood. They released them to the public on GCP only. Nvidia's cards are released to the public as a hardware device that you can customize around. Big difference.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#42

Post by baybal2 » Thu, Apr 26, 2018, 3:22 PM UTC

dgacmu wrote: post #31

samfisher83 wrote: post #23

>For GPUs, there are further interesting options to consider next to buying. For example, Cirrascale offers monthly rentals of a server with four V100 GPUs for around $7.5k (~$10.3 per hour). However, further benchmarks are required to allow a direct comparison since the hardware differs from that on AWS (type of CPU, memory, NVLink support etc.). Can't you just buy some 1080s for cheaper than this. I understand ther…

Yes, you can. The problem starts when "you" are a large company -- NVidia restricts "datacenter" use of consumer GPUs (see previous HN discussion of that one: https://news.ycombinator.com/item?id=15983587 ). A single Titan V is somewhere in the 90% range of a V100 at less than 1/3 the cost, and a 1080ti, if you can find one, likely offers a slightly better price/performance spot. 4-GPU training may suffer due the lac…

Hire people to buy 1080 in retail. This problem is solvable easily.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#43

Post by elmarhaussmann » Thu, Apr 26, 2018, 3:26 PM UTC

Jabbles wrote: post #39

elmarhaussmann wrote: post #2

Hi, author here. The motivation for this article came out of the HN discussion on a previous post ( https://news.ycombinator.com/item?id=16447096 ). There was a lot of valuable feedback - thanks for that. Happy to answer questions!

Do you have more information about this bit? the TPU implementation applies very compute-intensive image pre-processing steps and actually sacrifices raw throughput Thanks

In general, you try to keep the TPU/GPU busy 100%, so enough data needs to be readily accessible at any point in time. In this example, images needs to be read from disk, decoded, transformed (cropped, resized, normalized etc.) before they can be fed to the TPU. The transformations can be computationally intensive so they actually become a bottleneck.

In terms of how much compute power the TPU pre-processing needs I only have very rough numbers: I ran the same pre-processing while training ResNet-50 on a node with 4 GPUs and it was consistently utilizing >22 CPU cores (including all of the other CPU-tasks while training).

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#44

Post by shaklee3 » Thu, Apr 26, 2018, 3:27 PM UTC

elmarhaussmann wrote: post #33

Tenoke wrote: post #29

>For the V100 experiments, we used a p3.8xlarge instance ( Xeon E5–2686@2.30GHz 16 cores, 244 GB memory , Ubuntu 16.04) on AWS with four V100 GPUs (16 GB of memory each). For the TPU experiments, we used a small n1-standard-4 instance as host ( Xeon@2.3GHz two cores, 15 GB memory , Debian 9) for which we provisioned a Cloud TPU (v2–8) consisting of four TPUv2 chips (16 GB of memory each). A bit odd that the TPUs are…

All of the computation, including pre-processing, is offloaded to the TPU. The weak machine is really just idling. A bigger one will only cost money and have no measurable effect on the performance.

This is why my previous comment mentioned that GCP is a better benchmark for this since you can select the number of CPUs to match with the GPUs to some extent. You can get a rough idea of the savings by looking at their P100 instances.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#45

Post by shaklee3 » Thu, Apr 26, 2018, 3:31 PM UTC

jacksmith21006 wrote: post #32

Nokinside wrote: post #21

Nvidia is currently in cashing out phase. They have monopoly and money flows in effortlessly. The cost performance ratio reflects this. AMD will enter the game soon once they get their software working, Intel will follow. I suspect that Nvidia will respond with its own specialized machine learning and inference chips to match the cost/performance ratio. As long as Nvidia can maintain high manufacturing volumes and sm…

"The cost performance ratio reflects this." But the TPUs are half the cost per this article? Plus Google does the entire stack and can better optimize the hardware versus Nvidia. So it seem Google can improve faster I would think. If there ever was a huge advantage doing the entire stack it is with neural networks. A perfect example is Google new speech doing 16k samples a second with a NN. https://cloudplatform.goog…

I'm not sure what you mean by google does the entire stack. Nvidia writes all of the major CUDA libraries used behind the scenes in the NN libraries, such as cuDNN, cuBLAS, etc. Nvidia can likely improve their hardware significantly faster/more efficiently than Google can because their entire business depends on it. Google has incentive for improving their TPU for internal use, but they don't make any money by selling TPU time on GCP yet.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#46

Post by jacksmith21006 » Thu, Apr 26, 2018, 3:36 PM UTC

shaklee3 wrote: post #41

jacksmith21006 wrote: post #28

Earlier quoted context omitted.

That is true. But the cost of running is where all the cost is at really not so much in making the chips. Yes I can say it is a lot cheaper. That is what this article is all about. You can do about twice the images per dollar using the TPUs with GCP versus using Nvidia with AWS. Or what am I missing? BTW, Google has released to the general public. What are you talking about? "Google’s AI chips are now open for public…

You misunderstood. They released them to the public on GCP only . Nvidia's cards are released to the public as a hardware device that you can customize around. Big difference.

Yes in the cloud as you would expect in 2018. Available to the general public.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#47

Post by jacksmith21006 » Thu, Apr 26, 2018, 3:37 PM UTC

shaklee3 wrote: post #45

jacksmith21006 wrote: post #32

Earlier quoted context omitted.

"The cost performance ratio reflects this." But the TPUs are half the cost per this article? Plus Google does the entire stack and can better optimize the hardware versus Nvidia. So it seem Google can improve faster I would think. If there ever was a huge advantage doing the entire stack it is with neural networks. A perfect example is Google new speech doing 16k samples a second with a NN. https://cloudplatform.goog…

I'm not sure what you mean by google does the entire stack. Nvidia writes all of the major CUDA libraries used behind the scenes in the NN libraries, such as cuDNN, cuBLAS, etc. Nvidia can likely improve their hardware significantly faster/more efficiently than Google can because their entire business depends on it. Google has incentive for improving their TPU for internal use, but they don't make any money by sellin…

Google does the applications at scale and then each layer below and a big one is TF. A great example is the recent release of the new text to speech using NN.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#48

Post by jacksmith21006 » Thu, Apr 26, 2018, 3:39 PM UTC

jacksmith21006 wrote: post #47

shaklee3 wrote: post #45

Earlier quoted context omitted.

I'm not sure what you mean by google does the entire stack. Nvidia writes all of the major CUDA libraries used behind the scenes in the NN libraries, such as cuDNN, cuBLAS, etc. Nvidia can likely improve their hardware significantly faster/more efficiently than Google can because their entire business depends on it. Google has incentive for improving their TPU for internal use, but they don't make any money by sellin…

Google does the applications at scale and then each layer below and a big one is TF. A great example is the recent release of the new text to speech using NN.

When you use a Google service that uses the TPUs they are indirectly selling the TPUs.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#49

Post by modeless » Thu, Apr 26, 2018, 3:50 PM UTC

samfisher83 wrote: post #23

>For GPUs, there are further interesting options to consider next to buying. For example, Cirrascale offers monthly rentals of a server with four V100 GPUs for around $7.5k (~$10.3 per hour). However, further benchmarks are required to allow a direct comparison since the hardware differs from that on AWS (type of CPU, memory, NVLink support etc.). Can't you just buy some 1080s for cheaper than this. I understand ther…

1080s don't have the "tensor cores" of V100, or NVLink, so they will not get anywhere near the same performance on this benchmark.

Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50

#50

Post by eanzenberg » Thu, Apr 26, 2018, 3:54 PM UTC

jacksmith21006 wrote: post #4

Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…

The impression I got was opposite: TPU is not the hot shit that Google claims it is. Pricing is kind of irrelevant since they can subsidize this to create that story.