Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which GPU(s) to Get for Deep Learning

timdettmers.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Which GPU(s) to Get for Deep Learning

#71

Post by throwaway32131 »

zeptomu wrote: post #27

There is an interesting offer by Hetzner that's available for some month now. They provide a dedicated server with a GTX 1080 for ~99e/month (111$/month) with adequate CPU (i7-6700), 64G Memory, 500G disk space and 50TB Traffic - there are also on-demand offerings by GCP and AWS, but I do not think they can match the offer by Hetzner: https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ss... . Keep in mind that…

The electricity will likely cost a significant chunk of that, if one were running that full time in Germany.

0.5 * 24 * 30 * (30 c/kWH) = $108

However, it'd be a third of that in the states (may be cheaper in WA).

Re: Which GPU(s) to Get for Deep Learning

#72

Post by agibsonccc »

boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

Disclosure: My incentives compete with google cloud. We have had enormous cost savings with on premise customers. I think one thing that isn't being said here is: Most enterprise customers can't actually leverage that much GPU capacity anyways. We have found incremental addition of GPUs to hadoop clusters (yes this is a thing) to be great.

It's cheaper, allows gradual adoption of deep learning and is a familiar toolset for folks already doing some sort of machine learning.

I won't comment on research (not our domain). Depreciation on hardware is pretty standard - that being said it also comes with established SKUs from dell,hp,cisco,.. with proper support.

Analytics clusters (while hard to manage) are fairly robust already to job failures. The cost savings just makes a ton more sense when you are doing continuous workloads for different use cases.

Re: Which GPU(s) to Get for Deep Learning

#73

Post by boulos »

tensor wrote: post #68
boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.

Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000.

I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and they're really expensive ;).

Re: Which GPU(s) to Get for Deep Learning

#74

Post by throwaway32131 »

redtuesday wrote: post #25
jbmorgado wrote: post #21

Earlier quoted context omitted.

The question is not AMD's processing power, the question is the tools available for AMD cards, which are scant to say the best. While AMD doesn't create something like cuDNN to go along with their cards, no real work will start being done in porting most important DL libraries to AMD cards. And even in that case, it will be a lost generation for AMD, only in the 2nd generation where AMD actually offers a real alterna…

They stated at the analyst day that they will support TensorFlow, Theano, Caffee etc. And will release something called MiOpen. They also stated this last year: > On top of ROCm, deep-learning developers will soon have the opportunity to use a new open-source library of deep learning functions called MIOpen that AMD intends to release in the first quarter of next year. This library offers a range of functions pre-opt…

They don't have an implementation of in-place convolutions AFAIK. The im2col trick may be too slow in practice (read x2.5-10 slower) to make any difference in terms of adoption. We already see this with Caffe and cl-Torch.

Re: Which GPU(s) to Get for Deep Learning

#76

Post by zo7 »

boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

Except buying hardware isn't a 24-36 month investment, you could buy new hardware every 3 months or so and it would still be cheaper than renting. Renting only makes sense when you need to scale horizontally, but for most people's use cases having an in-house machine is sufficient.

Re: Which GPU(s) to Get for Deep Learning

#77

Post by boulos »

agibsonccc wrote: post #72
boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

Disclosure: My incentives compete with google cloud. We have had enormous cost savings with on premise customers. I think one thing that isn't being said here is: Most enterprise customers can't actually leverage that much GPU capacity anyways. We have found incremental addition of GPUs to hadoop clusters (yes this is a thing) to be great. It's cheaper, allows gradual adoption of deep learning and is a familiar tools…

While it's easy enough to add GPUs to a Hadoop/Spark cluster (and we did so too via Dataproc [1]) are you just saying that means you assume closer to 100% utilization due to sharing?

If so, that's fine-ish, but then people have to wait (you're either full and people are waiting or you're at less than 100%). My preference is to run for XX minutes per job on-demand (per person). If you have tons of non-overlapping users, you can absolutely aggregate them. But how many do you buy and how quickly do you upgrade to newer hardware?

[1] https://cloud.google.com/dataproc/docs/concepts/gpu-clusters

Re: Which GPU(s) to Get for Deep Learning

#78

Post by tanilama »

boulos wrote: post #73
tensor wrote: post #68

Earlier quoted context omitted.

The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.

Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and t…

A GTX 1080 is like $550 at this point, K80 cannot really compete with the cost effectiveness here. And it is probably not going out of fashion in one year, so the price is totally worth it.

The real attractiveness for cloud at this point is if you are going to train your model with 8-GPU or more, that is likely not feasible for individual enthusiasts, but demand for such machine is rare for hobbyists anyway.

Re: Which GPU(s) to Get for Deep Learning

#79

Post by zo7 »

boulos wrote: post #73
tensor wrote: post #68

Earlier quoted context omitted.

The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.

Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and t…

Isn't there an abstraction cost though, since that price is only for half a board? In my experience (with AWS) I've had a machine with a Titan X outrun a cloud instance by 1.5-2x, which is significant.

There are also other hidden costs that aren't factored into that number. I'm having a hard time getting anything reasonable for under $750/month for 24/7 usage: https://cloud.google.com/products/calculator

Re: Which GPU(s) to Get for Deep Learning

#80

Post by netheril96 »

KaoruAoiShiho wrote: post #56
Sephr wrote: post #43

Here's a much more cost effective answer: Use https://cloud.google.com/tpu/ Unless you have an unlimited supply of free electricity and don't care about the increased hardware management overhead, it's a waste of money to buy Pascal GPUs for large-scale deep learning. The following cards have much more optimized deep learning silicon and are publicly available /right now/: - Nvidia Tesla V100 (Tensor cores only: 120…

That makes no sense as the TPU2 is not really out yet for actual consumers. AFAIK it's only in closed alpha right now, so if you're actually doing stuff right now it's not a real option. There's also no pricing so the "cost effective" remark can't really be seen.

If and when it is for sale, I wonder how many problems with the hardware and drivers customers will face over the years.
Post reply on HN