Slower alternative: "fastai with @pytorch on @awscloud is currently the fastest to train Imagenet on GPU, fastest on a single machine (faster than Intel-caffe on 64 machines!), and fastest on public infrastructure (faster than @TensorFlow on a TPU!)
Big thanks to our students that helped with this." - https://twitter.com/jeremyphoward/status/988852083796291584
Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
51–60 of 132 posts
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#52>For the V100 experiments, we used a p3.8xlarge instance ( Xeon E5–2686@2.30GHz 16 cores, 244 GB memory , Ubuntu 16.04) on AWS with four V100 GPUs (16 GB of memory each). For the TPU experiments, we used a small n1-standard-4 instance as host ( Xeon@2.3GHz two cores, 15 GB memory , Debian 9) for which we provisioned a Cloud TPU (v2–8) consisting of four TPUv2 chips (16 GB of memory each). A bit odd that the TPUs are…
All of the computation, including pre-processing, is offloaded to the TPU. The weak machine is really just idling. A bigger one will only cost money and have no measurable effect on the performance.
What is the cost difference between the CPUs on the google cloud vs AWS? How would adjusting for it effect the cost/images ratio?
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#53Hi, author here. The motivation for this article came out of the HN discussion on a previous post ( https://news.ycombinator.com/item?id=16447096 ). There was a lot of valuable feedback - thanks for that. Happy to answer questions!
Don't TPUs get sustained use discounts? I know they're not preemptible. That would be comparable to AWS reserved instances.
EDIT: you don't get sustained use discounts, either, at the moment. You can get either for GCP GPUs, though. Perhaps that will change once TPUs are out of beta?
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#54>For the V100 experiments, we used a p3.8xlarge instance ( Xeon E5–2686@2.30GHz 16 cores, 244 GB memory , Ubuntu 16.04) on AWS with four V100 GPUs (16 GB of memory each). For the TPU experiments, we used a small n1-standard-4 instance as host ( Xeon@2.3GHz two cores, 15 GB memory , Debian 9) for which we provisioned a Cloud TPU (v2–8) consisting of four TPUv2 chips (16 GB of memory each). A bit odd that the TPUs are…
The TPU is not really just the chip. It has an actual machine that is provisioned behind the scenes and accepts RPC calls. Good luck finding out its specs. All you're supposed to care about are the address and port it answers at.
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#55Earlier quoted context omitted.
You misunderstood. They released them to the public on GCP only . Nvidia's cards are released to the public as a hardware device that you can customize around. Big difference.
Yes in the cloud as you would expect in 2018. Available to the general public.
They announced in 2016 they had TPUs. So no, I would not expect that 2 full years later they're just now being available in the public cloud. These are not new products to them; they likely just don't want to deal with supporting them in different configurations.
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#56Hi, author here. The motivation for this article came out of the HN discussion on a previous post ( https://news.ycombinator.com/item?id=16447096 ). There was a lot of valuable feedback - thanks for that. Happy to answer questions!
"As shown above, the top-1 accuracy after 90 epochs for the TPU implementation is 0.7% better. This may seem minor, but making improvements at this already very high level is extremely difficult and, depending on the application, such small improvements may make a big difference in the end."
Any idea of how much variation in accuracy you get on different training runs of the same model on the same hardware? My understanding is that model quality can and does vary from one run to the next on these kinds of large datasets - from a single observation, it's hard to know if the difference is real or noise.
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#57Nvidia is currently in cashing out phase. They have monopoly and money flows in effortlessly. The cost performance ratio reflects this. AMD will enter the game soon once they get their software working, Intel will follow. I suspect that Nvidia will respond with its own specialized machine learning and inference chips to match the cost/performance ratio. As long as Nvidia can maintain high manufacturing volumes and sm…
"The cost performance ratio reflects this." But the TPUs are half the cost per this article? Plus Google does the entire stack and can better optimize the hardware versus Nvidia. So it seem Google can improve faster I would think. If there ever was a huge advantage doing the entire stack it is with neural networks. A perfect example is Google new speech doing 16k samples a second with a NN. https://cloudplatform.goog…
> But the TPUs are half the cost per this article?
Exactly. Nvidia can match the performance already without 100% specialized processor. It's the just the price they need to cut by optimizing their architecture for tensor processing and reducing their profits when competition emerges.
Google is not in the business of becoming a major chip maker or competing with Nvidia head on. Putting hundreds of millions into new microarchitecture every second year eats lots of resources. They just want competitive market and the prices to go down.
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#58Excellent work. Do you have plans to open source the scripts/implementation details used to reproduce the results? Would be great if others can also validate and repeat the experiment for future software updates (e.g. TensorFlow 1.8) as I expect there will be some performance gain for both TPU and GPU by CUDA and TensorFlow optimizations.
Sidenote: Love the illustrations that accompany most of your blog posts, are they drawn by an in-house artist/designer?
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#59Hi, author here. The motivation for this article came out of the HN discussion on a previous post ( https://news.ycombinator.com/item?id=16447096 ). There was a lot of valuable feedback - thanks for that. Happy to answer questions!
"As shown above, the top-1 accuracy after 90 epochs for the TPU implementation is 0.7% better. This may seem minor, but making improvements at this already very high level is extremely difficult and, depending on the application, such small improvements may make a big difference in the end." Any idea of how much variation in accuracy you get on different training runs of the same model on the same hardware? My unders…
I've been running a lot of these resnet-50 experiments lately and the run-to-run variation is very small, on the order of 0.1%. It's actually pretty amazing how consistent training is given that the initialization is always different and the data is sampled differently on each run. (As an aside, it took us about three weeks to track down a bug that was causing the model to consistently reach an accuracy 1% lower than it was supposed to.)
Re: Comparing Google’s TPUv2 against Nvidia’s V100 on ResNet-50
#60Thanks for sharing and very insightful. Guess the TPUs are the real deal. About 1/2 the cost for similar performance. Would assume Google is able to do that because of the less power required. I am actually more curious to get a paper on the new speech NN Google is using. Suppose to be 16k samples a second through a NN is hard to imagine how they did that and was able to roll it out as you would think the cost would…
Half the cost? Where are you reading that? Yeah on demand rental in AWS is expensive, but both long term and buying V100 yourself is significantly cheaper. Cloud companies have pretty fat margins on on demand rentals. You can’t buy a TPU, it’s a cloud only thing. They also show it’s not a huge difference in both perf and time to converge (albeit only one architecture) I would say kudos to V100 and this benchmark that…
See the chart titled: Performance in images per second per $.
TPUv2 is has 1.27x-1.86x the images/s/$.
And the other chart titled: Cost to reach 75.7% top-1 accuracy.
Where TPUv2 costs 62.5% the reserved GPU instance and 42.6% the unreserved GPU cost.
Key takeaway from the article:
> While the V100s perform similarly fast, the higher price and slower convergence of the implementation results in a considerably higher cost-to-solution.