Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

minimaxir.com

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#101

Post by 0xbear »

beagle3 wrote: post #99
0xbear wrote: post #97

Earlier quoted context omitted.

I have never encountered such an abnormal workload. This is also less likely to happen in Broadwell Xeon and up, where last level cache can be partitioned. And this is also less likely to happen on Google Cloud in particular, because Google uses high end CPUs with tons of cache.

If both core threads are memory (and cache) intensive, then you get effectively half the cache size and half the memory bandwidth. Partitioning may make eviction less random, but the cache size is still halved, regardless of how much "tons of cache" you start with.

Increasing cache has the net effect of increasing hit ratio, sometimes substantially. With 20MB per die this may change the calculation of where things drop off. I have found that I can't reliably predict how a chip will perform, so I just wrote a bunch of benchmarks and it takes me about half an hour to see if the chip performs better or worse than I thought it would. Google's Broadwell VMs perform very well.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#102

Post by boulos »

boulos wrote: post #72
asah wrote: post #63

Earlier quoted context omitted.

Just signed up and ported my model + data: - it's indeed noticeably faster than the Google VMs. As usual, I compiled tensorflow for this GPU vs K80 (feature 6.1 vs 3.7). - ubuntu 16 minimal is indeed "minimal" ! but it worked... - GTX 1080 (7.92GB) has less GPU RAM than the K80 (11.17GiB) - this required me to reduce the model design slightly. For my model/data, Hetzner runs 1 training epoch in 1 hr vs 1.75 hr for Go…

This is presumably just the full board versus half nomenclature noted above. But yes, consumer GPUs are way more cost competitive than Tesla class parts. Being able to train bigger models is valuable to some folks, but not everyone, so I don't begrudge using the GTX line. Disclosure: I work on Google Cloud.

(Also, I can no longer edit, but a colleague pointed out that I should have read more carefully. A GTX 1080 is a Pascal part, which compared to the poor old Kepler in K80s, it'll really shine. Volta all the more so in the next year).

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#103

Post by nevi-me »

ryanlol wrote: post #86
nevi-me wrote: post #67

Earlier quoted context omitted.

Wow, to go off topic; I've been using Versaweb for the past 3-4 years, but became very unhappy after they forced a "server management" fee down our throats. We've been looking at setting up a small cluster of servers at work (budget of about $500), and I was still going to go with Versaweb. After seeing Hetzner, I'm going to reassess, and likely move everything there. I'm paying €150 for what it seems I could pay €10…

Why are you comparing Hetzner and Versaweb? They exist in completely different markets.

How so? I understand that there are some differences between the two, but the fundamentals are the same; which is that I want a physical server that I can manage.

Their pricing structures are slightly different, Versaweb gives me a bit more flexibility when configuring, a wider IP subnet bundled (instead of 1 usable IP), and a few other things which I'm investigating.

I also have to consider laws and network latency as these are in different regions.

In the end, I am paying $180 for a Haswell Xeon with lots of disk space and IO. I could pay the same amount for more RAM on the same CPU, albeit with slightly less space.

If I keep the same setup at a fraction of the cost, I could end up getting the 1080 GPU on the same datacenter. It somehow feels like the same or similar market to my needs ...

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#104

Post by barrus »

paulsutter wrote: post #2

Shoutout for Hetzner's 99 euro/month server with a GTX 1080, much better than the pseudo-K80s that Google Cloud provides for $520/month. The Google K80s are half or quarter the speed of a real K80, part of the reason they show so badly in the comparison. https://www.hetzner.com/dedicated-rootserver/ex51-ssd-gpu?co...

Looked into this a bit more. The GTX 1080 is based on the Pascal architecture and so will be faster than any Kepler-based K80 on any cloud - even faster than a K80 card with 2 GPUs. The GTX is a consumer board and is less expensive than the datacenter equivalent P100 PCIe card. The P100 has 16 GB ram and HBM2 memory (twice the memory and more than twice the memory bandwidth) and supports ECC if you care about detecting memory corruption. The P100 will be faster than the GTX 1080 once it is available. As I said before, GCP offers K80 GPUs in passthrough mode and you can use a single K80 die ($0.70 / hour billed by the minute) or you can attach up to 8 K80 GPUs to a single VM. Disclaimer: I am a product manager for GPUs in Google Cloud.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#105

Post by thesandlord »

ec109685 wrote: post #65
thesandlord wrote: post #19

Earlier quoted context omitted.

Google Cloud Developer Advocate here. Go for it as long as you understand the downside. It's possible that all instances get preempted at once (especially at the 24hr mark), that there isn't capacity to spin up new preemptible nodes in the selected zone once the old instance is deleted, etc. New VMs also take time to boot and join the cluster. If you are just doing dev/test stuff, I'd recommend using a namespace in y…

What about spreading your K8s load across multiple instance types (given it is unlikely google runs out of all types at the same time). That plus historical modeling was the trick of a startup that Amazon acquired that promised to dramatically reduce compute cost, by using mostly spot instances. Would those types of mitigations work similarly with Google's premetable VM's?

Compute Engine doesn't really have "Instance Types" or "Instance Families" per say, just Core/Memory combinations. Larger machines have a higher chance of preemption though (according to the PM of PVMs who stated that on this thread).

There are a few interesting projects out there that do the kind of automation you are speaking of like these:

https://github.com/binary-com/gce-manager https://github.com/skelterjohn/prevmtable

Spreading multiple smaller machines over a multi-zone k8s deployment might help mitigate, but it will never solve all the issues.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#106

Post by ryanlol »

nevi-me wrote: post #103
ryanlol wrote: post #86

Earlier quoted context omitted.

Why are you comparing Hetzner and Versaweb? They exist in completely different markets.

How so? I understand that there are some differences between the two, but the fundamentals are the same; which is that I want a physical server that I can manage. Their pricing structures are slightly different, Versaweb gives me a bit more flexibility when configuring, a wider IP subnet bundled (instead of 1 usable IP), and a few other things which I'm investigating. I also have to consider laws and network latency…

>How so? I understand that there are some differences between the two, but the fundamentals are the same; which is that I want a physical server that I can manage.

They're on different continents, which is a pretty fundamental difference.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#107

Post by paulsutter »

barrus wrote: post #104
paulsutter wrote: post #2

Shoutout for Hetzner's 99 euro/month server with a GTX 1080, much better than the pseudo-K80s that Google Cloud provides for $520/month. The Google K80s are half or quarter the speed of a real K80, part of the reason they show so badly in the comparison. https://www.hetzner.com/dedicated-rootserver/ex51-ssd-gpu?co...

Looked into this a bit more. The GTX 1080 is based on the Pascal architecture and so will be faster than any Kepler-based K80 on any cloud - even faster than a K80 card with 2 GPUs. The GTX is a consumer board and is less expensive than the datacenter equivalent P100 PCIe card. The P100 has 16 GB ram and HBM2 memory (twice the memory and more than twice the memory bandwidth) and supports ECC if you care about detecti…

The P100 is about 10x the price of the 1080 ($6000-9000 vs $500 for the 1080 and $700 for the 1080TI).

I've talked with several second-tier cloud providers, and the GTX 1080TI is what their large-deployment customers use. At the NVIDIA conference they were all promoting the P100 (NVIDIA insisted), but all admitted that nobody asked them to deploy P100s at scale.

The Hetzner box is about 0.15 an hour. That means more GPUs per developer.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#108

Post by LeaderGPU »

Thank you for benchmarks! It will be interesting include in your research Inception-v4, Inception-ResNet. And try to compute with Nvidia 1080 / 1080TI cards.

Our benchmarks for processing 1000000 images ResNet-50:

- 8x Tesla K80: 43m 3 sec.

- 8x Nvidia 1080: 17m 32 sec ( 0.09 euro / minute ).

We can provide you resources for free for research.

Disclosure: I'm founder of LeaderGPU.

Post reply on HN