Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

minimaxir.com

Post reply on HN
Print view View original

21–30 of 108 posts

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#21

Post by swampthinker »

thesandlord wrote: post #19
ohstopitu wrote: post #18

Earlier quoted context omitted.

something I've always been curious about (and if a Google Cloud Engineer could clear up - that would be great), is why we should not (as in, why does everyone not) use preemptible nodes (apart from maybe the 3 / 5 master nodes). My question specifically being: if I configure a k8s cluster to have all my slaves as preemptible nodes...would GCP automatically add new nodes as my old nodes are deleted (from what I unders…

Google Cloud Developer Advocate here. Go for it as long as you understand the downside. It's possible that all instances get preempted at once (especially at the 24hr mark), that there isn't capacity to spin up new preemptible nodes in the selected zone once the old instance is deleted, etc. New VMs also take time to boot and join the cluster. If you are just doing dev/test stuff, I'd recommend using a namespace in y…

If you don't mind me asking, what exactly is the role of a developer advocate?

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#22

Post by shusson »

barrus wrote: post #20
shusson wrote: post #12

Earlier quoted context omitted.

> Google K80s are half or quarter the speed of a real K80 do you have any evidence for this?

Each Google K80s is one GPU or 1/2 of a K80 board, so technically you are correct that a Google K80 GPU is half of a K80 board. However, they are offered in passthrough mode and achieve full performance. If you want a whole K80 board, attach 2 K80 GPUs to a single VM. You can have 1, 2, 4 or 8 K80 GPUs attached to each VM in GCP. (I'm one of the GPU product managers at Google Cloud).

thanks for the info. On a side note, do you know if GCP will ever support preemptible GPU instances?

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#23

Post by paulsutter »

barrus wrote: post #20
shusson wrote: post #12

Earlier quoted context omitted.

> Google K80s are half or quarter the speed of a real K80 do you have any evidence for this?

Each Google K80s is one GPU or 1/2 of a K80 board, so technically you are correct that a Google K80 GPU is half of a K80 board. However, they are offered in passthrough mode and achieve full performance. If you want a whole K80 board, attach 2 K80 GPUs to a single VM. You can have 1, 2, 4 or 8 K80 GPUs attached to each VM in GCP. (I'm one of the GPU product managers at Google Cloud).

It would be polite to indicate that on the price list (understatement).

While you're here: the other reason we switched to Hetzner is reliability. Sure we can continue training from the last checkpoint but we still lost half a day on average for the many surprise reboots. We suspect that you've overbooked the GPUs and someone has to lose when too many connect.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#24

Post by gruez »

0xbear wrote: post #13

FYI, y'all: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator.

>FYI, y'all: cloud "cores" are actually hyperthreads

Depends on the provider. Azure, for instance, has hyperthreading disabled on most of their configurations. They're starting to offer new configurations with hyperthreading though.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#25

Post by tgtweak »

0xbear wrote: post #13

FYI, y'all: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator.

This is very important if you're running any cpu intensive workload at scale. We had custom compiled x264 then custom compiled that into ffmpeg to get everything out of our CPUs for an encoding cluster. AMD cpus seem to really shine here.

You'd be surprised the difference it makes. It was one of the reasons I liked Gentoo, emerge would always build from source for your target CPU flags, instead of using the package managed "one size fits all" build. Those 5-10%s really compound when you add them up along all dependencies.

Kudos to tutorials and guides that instruct how to build from source.

The same is every bit as true today for your containers, assuming you have a homogeneous target to run them (yes I know, containers are supposed to be supremely portable, but private ones can be purpose built)

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#26

Post by 0xbear »

gruez wrote: post #24
0xbear wrote: post #13

FYI, y'all: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator.

>FYI, y'all: cloud "cores" are actually hyperthreads Depends on the provider. Azure, for instance, has hyperthreading disabled on most of their configurations. They're starting to offer new configurations with hyperthreading though.

Yep. But they compensate for that by charging a lot more and using lower end CPU SKUs with less cache. And GPUs are still per die.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#27

Post by chris_st »

Quick question to those with a deep understanding of these things... I have not been able to get GPU tensor flow (on AWS) to run faster for the networks I'm using.

This is with a small(ish) network of perhaps a few hundred nodes... should I see a speedup for this case, or are GPUs only relevant for large CNNs, etc.?

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#28

Post by shmageggy »

paulsutter wrote: post #2

Shoutout for Hetzner's 99 euro/month server with a GTX 1080, much better than the pseudo-K80s that Google Cloud provides for $520/month. The Google K80s are half or quarter the speed of a real K80, part of the reason they show so badly in the comparison. https://www.hetzner.com/dedicated-rootserver/ex51-ssd-gpu?co...

Wish HN had a 'save' feature so I can remember this comment when I need a GPU box.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#29

Post by 0xbear »

tgtweak wrote: post #25
0xbear wrote: post #13

FYI, y'all: cloud "cores" are actually hyperthreads. Cloud GPUs are single dies on multi-die card. If you use GPUs 24x7, just buy a few 1080 Ti cards and forego the cloud entirely. If you must use TF in cloud with CPU, compile it yourself with AVX2 and FMA support. Stock TF is compiled for the lowest common denominator.

This is very important if you're running any cpu intensive workload at scale. We had custom compiled x264 then custom compiled that into ffmpeg to get everything out of our CPUs for an encoding cluster. AMD cpus seem to really shine here. You'd be surprised the difference it makes. It was one of the reasons I liked Gentoo, emerge would always build from source for your target CPU flags, instead of using the package m…

This is especially important if most of your workload is matrix multiplication. Those workloads heavily benefit from vectorization. It might also help to enable Intel MKL, because Eigen, which TF uses by default is not the fastest thing out there, just the most convenient to work with cross platform.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#30

Post by david-gpu »

chris_st wrote: post #27

Quick question to those with a deep understanding of these things... I have not been able to get GPU tensor flow (on AWS) to run faster for the networks I'm using. This is with a small(ish) network of perhaps a few hundred nodes... should I see a speedup for this case, or are GPUs only relevant for large CNNs, etc.?

Correct. GPUs are not efficient with very small networks.
Post reply on HN