Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

minimaxir.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#31

Post by girvo »

shmageggy wrote: post #28
paulsutter wrote: post #2

Shoutout for Hetzner's 99 euro/month server with a GTX 1080, much better than the pseudo-K80s that Google Cloud provides for $520/month. The Google K80s are half or quarter the speed of a real K80, part of the reason they show so badly in the comparison. https://www.hetzner.com/dedicated-rootserver/ex51-ssd-gpu?co...

Wish HN had a 'save' feature so I can remember this comment when I need a GPU box.

If you upvote it, you can find it again through your profile. I'm on mobile but I think you can favourite comments too via the time stamp link, also shows on your profile

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#32

Post by deepnotderp »

chris_st wrote: post #27

Quick question to those with a deep understanding of these things... I have not been able to get GPU tensor flow (on AWS) to run faster for the networks I'm using. This is with a small(ish) network of perhaps a few hundred nodes... should I see a speedup for this case, or are GPUs only relevant for large CNNs, etc.?

In theory, there's no reason that a GPU shouldn't be faster.

In practice, there's a multitude of reasons why CPUs are more efficient (or at least faster) for smaller networks.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#33

Post by edwinksl »

0xbear wrote: post #29
tgtweak wrote: post #25

Earlier quoted context omitted.

This is very important if you're running any cpu intensive workload at scale. We had custom compiled x264 then custom compiled that into ffmpeg to get everything out of our CPUs for an encoding cluster. AMD cpus seem to really shine here. You'd be surprised the difference it makes. It was one of the reasons I liked Gentoo, emerge would always build from source for your target CPU flags, instead of using the package m…

This is especially important if most of your workload is matrix multiplication. Those workloads heavily benefit from vectorization. It might also help to enable Intel MKL, because Eigen, which TF uses by default is not the fastest thing out there, just the most convenient to work with cross platform.

Would hyperthreading be helpful or harmful?

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#34

Post by chris_st »

david-gpu wrote: post #30
chris_st wrote: post #27

Quick question to those with a deep understanding of these things... I have not been able to get GPU tensor flow (on AWS) to run faster for the networks I'm using. This is with a small(ish) network of perhaps a few hundred nodes... should I see a speedup for this case, or are GPUs only relevant for large CNNs, etc.?

Correct. GPUs are not efficient with very small networks.

Any useful rules of thumb on when to use GPU?

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#35

Post by 0xbear »

edwinksl wrote: post #33
0xbear wrote: post #29

Earlier quoted context omitted.

This is especially important if most of your workload is matrix multiplication. Those workloads heavily benefit from vectorization. It might also help to enable Intel MKL, because Eigen, which TF uses by default is not the fastest thing out there, just the most convenient to work with cross platform.

Would hyperthreading be helpful or harmful?

Hyper threading is not harmful per se. It lets your CPU make forward progress when it would otherwise be stalled waiting for something. My issue is that they call hyperthreads "vCPU" which makes it seem like you're getting a full core, while in reality you're getting 60% of a core at most.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#36

Post by jsmthrowaway »

0xbear wrote: post #35
edwinksl wrote: post #33

Earlier quoted context omitted.

Would hyperthreading be helpful or harmful?

Hyper threading is not harmful per se. It lets your CPU make forward progress when it would otherwise be stalled waiting for something. My issue is that they call hyperthreads "vCPU" which makes it seem like you're getting a full core, while in reality you're getting 60% of a core at most.

vCPU is a different concept than hyperthreading logical cores, though. They're decoupled. (vCPU comes from virtualization software like Xen.)

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#37

Post by minimaxir »

paulsutter wrote: post #23
barrus wrote: post #20

Earlier quoted context omitted.

Each Google K80s is one GPU or 1/2 of a K80 board, so technically you are correct that a Google K80 GPU is half of a K80 board. However, they are offered in passthrough mode and achieve full performance. If you want a whole K80 board, attach 2 K80 GPUs to a single VM. You can have 1, 2, 4 or 8 K80 GPUs attached to each VM in GCP. (I'm one of the GPU product managers at Google Cloud).

It would be polite to indicate that on the price list (understatement). While you're here: the other reason we switched to Hetzner is reliability. Sure we can continue training from the last checkpoint but we still lost half a day on average for the many surprise reboots. We suspect that you've overbooked the GPUs and someone has to lose when too many connect.

The price list indicates that 1 instance is 1/2 a device, and so on: https://cloud.google.com/compute/pricing#gpus

Although I agree it is somewhat confusing in terms of performance.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#38

Post by chewxy »

paulsutter wrote: post #2

Shoutout for Hetzner's 99 euro/month server with a GTX 1080, much better than the pseudo-K80s that Google Cloud provides for $520/month. The Google K80s are half or quarter the speed of a real K80, part of the reason they show so badly in the comparison. https://www.hetzner.com/dedicated-rootserver/ex51-ssd-gpu?co...

That's surprisingly affordable. Commenting to save

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#39

Post by chewxy »

chris_st wrote: post #27

Quick question to those with a deep understanding of these things... I have not been able to get GPU tensor flow (on AWS) to run faster for the networks I'm using. This is with a small(ish) network of perhaps a few hundred nodes... should I see a speedup for this case, or are GPUs only relevant for large CNNs, etc.?

It really depends on the type of things that you do - if your network is deep and has a lot of matrix multiplication, GPUs definitely do speed things up. Libraries like cuDNN have built in optimized convolution ops that will also make convolutions a lot faster.

In my experience (not tf related, I mainly work on my own library now: https://github.com/chewxy/gorgonia) even with a cgo penalty, deep networks do improve with GPU training. Never dabbled much in CNNs (convolutions tend to do my head in) so can't say much.

Re: Benchmarking TensorFlow on Cloud CPUs: Cheaper Deep Learning Than Cloud GPUs

#40

Post by fhoffa »

swampthinker wrote: post #21
thesandlord wrote: post #19

Earlier quoted context omitted.

Google Cloud Developer Advocate here. Go for it as long as you understand the downside. It's possible that all instances get preempted at once (especially at the 24hr mark), that there isn't capacity to spin up new preemptible nodes in the selected zone once the old instance is deleted, etc. New VMs also take time to boot and join the cluster. If you are just doing dev/test stuff, I'd recommend using a namespace in y…

If you don't mind me asking, what exactly is the role of a developer advocate?

https://medium.com/google-cloud/a-day-in-the-life-of-a-devel...

Not me or OP, but same team :)

Post reply on HN