Which GPU(s) to Get for Deep Learning • Hacker News Board

Re: Which GPU(s) to Get for Deep Learning

#81

Post by ktta » Tue, May 23, 2017, 3:50 AM UTC

solomatov wrote: post #52

ktta wrote: post #48

If anyone here doesn't want to spend money on $500+ GPU (or the $1k+ ones!), then I'd suggest getting the lowest tier Nvidia GPU for $100~[1] If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. So you can spend 10% of $1k every year and keep getting better…

>If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. GCP and AWS have old GPUs and they are really really expensive. If you expect to run workloads for a long time, it would be more cost efficient to buy your own hardware.

They are very, very few cases where getting an expensive GPU - anything more than a Nvidia 1060 would make financial sense.

The 1050 is a beginner card is it is perfectly fine to learn and run small nets. More importantly, you can decide if machine learning is for you. Then, comes to second investment, which is actually running real-world models.

Although the online GPU offering is expensive (you can also look around for cloud GPUs with lower SLA requirements for a lot cheaper) you'd be using them for a lot lesser time.

Even if you go with the big three, you can get good pricing if you look around. AWS has a single GPU with 1,536 CUDA cores and 4GB RAM. Looking at the Spot Pricing[1] which is $0.25 an hour, you can get 3000 hours of compute for $750, which is the price of 1080Ti (which is the most cost effective card on the market today).

Now, I would say, 3000 hours is more than enough time one needs to run whatever they want to. You can get a lot more if you go with some other service other than the big three.

If your usage exceeds that, then you are probably using it for commercial purposes, in which case you need to account for a lot more variables (downtime, maintenance, etc). Then you should also consider the deprecating perf/$ your GPU gives you every year over the new ones in the market - especially since the perf jump is a lot more than what we're seeing with CPUs.

If you are doing it for learning, I'd say you're doing something wrong, because you shouldn't need that much power. Also, consider the electricity costs and possible over usage of your computer.

[1]:https://aws.amazon.com/ec2/spot/pricing/ (check US East N. Virginia)

Re: Which GPU(s) to Get for Deep Learning

#82

Post by ktta » Tue, May 23, 2017, 3:58 AM UTC

throwaway91111 wrote: post #24

jonathanpoulter wrote: post #20

What impact does Google's Tensor Processing Unit have on the answer to this question?

Well, AFAIK, it isn't for sale; you have to go through their cloud offering. So it's not impactful at all on what GPU to get.

I mean it can have an effect on where the answer is to not get a GPU at all and use the cloud service.

Re: Which GPU(s) to Get for Deep Learning

#83

Post by ktta » Tue, May 23, 2017, 4:04 AM UTC

neurotech1 wrote: post #39

jonathanpoulter wrote: post #20

What impact does Google's Tensor Processing Unit have on the answer to this question?

AFAIK AMD has not announced any Tensor Processing Unit or specific tensor acceleration capability. nVidia Volta [0] does have a Tensor Processing core optimization. https://www.nvidia.com/en-us/data-center/volta-gpu-architect...

'Tensor processing' isn't some special kind of compute. The TPUs (which were badly named by google) are just specifically designed ASICs with lower precision which have much lower power consumption than GPUs, therefore cost effective for inference, which google does more than anyone on earth.

'Tensor Processing Unit' has become some what a phrase used to confuse (trick?) people into thinking it's some new type of processing, but its not. If the GPU says it has a Tensor Processing Unit, it just means it operates with a lower precision, but you have to remember that the overall power consumption of the GPU doesn't really change at full use compared to using higher precision chips. So you're actually missing out on the cost effectiveness of using lower precision.

If anything the 'Tensor Processing Units' just take up unnecessary die space when included high higher precision units because they're bad for training compared to higher precision compute units.

Re: Which GPU(s) to Get for Deep Learning

#84

Post by agibsonccc » Tue, May 23, 2017, 4:30 AM UTC

boulos wrote: post #77

agibsonccc wrote: post #72

Earlier quoted context omitted.

Disclosure: My incentives compete with google cloud. We have had enormous cost savings with on premise customers. I think one thing that isn't being said here is: Most enterprise customers can't actually leverage that much GPU capacity anyways. We have found incremental addition of GPUs to hadoop clusters (yes this is a thing) to be great. It's cheaper, allows gradual adoption of deep learning and is a familiar tools…

While it's easy enough to add GPUs to a Hadoop/Spark cluster (and we did so too via Dataproc [1]) are you just saying that means you assume closer to 100% utilization due to sharing? If so, that's fine-ish, but then people have to wait (you're either full and people are waiting or you're at less than 100%). My preference is to run for XX minutes per job on-demand (per person). If you have tons of non-overlapping user…

I'm talking about on prem clusters where people aren't using cloud. Many of these are just managed by central IT depts (eg: most of enterprise) . We've found folks are perfectly happy to add gpu nodes to an existing cluster managed by yarn or mesos. The incremental upgrade with just another SKU with an existing hardware vendor purchasing/procurement already works with is usually enough.

That being said - it can usually be every 6 months with a renewal of every 2 years or so. Your hardware timelines aren't far off. That being said - I was getting that upgrade cycles don't matter as much.

GPU clusters are enough of a new thing for enterprise yet that the jobs being run aren't even that high spec yet.

My opinion nothing more not claiming this is fact: Google has marched so far ahead of the rest of the world they aren't really paying attention to where current clusters and usage are. It seems a lot of the cloud usage is oriented towards startups and researchers (which isn't a bad thing, most folks in DL are researchers).

For enterprise, they might offload some workloads. There are definitely some workloads where cloud resources (spin up and shut down) make a ton of sense. Cloud servers are overly expensive otherwise.

Re: Which GPU(s) to Get for Deep Learning

#85

Post by tensor » Tue, May 23, 2017, 5:05 AM UTC

boulos wrote: post #73

tensor wrote: post #68

Earlier quoted context omitted.

The problem is that a month of GPU time on a cluster can buy you the hardware itself. If you are doing serious deep learning work it's just not cost effective at this point. If the costs come down by half or more, it may start looking viable for people who need a lot of resources.

Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and t…

Sure, in looking at what it would cost to replicate this:

https://arxiv.org/pdf/1704.01444.pdf

They say they used four Titan X cards for a month. I was actually looking at google cloud machine learning since I thought being targeted to tensorflow would be the most cost effective. But it's 0.49/hour per ML training unit, or 1.47/hour per GPU. A basic gpu gives 3 training units, so I thought that something approximately similar would require 12 training units, which comes to something like $4000+ a month. Maybe I completely misunderstood the resources being offered though, because you are right that cloud engine costs seem much lower.

I'll have to revisit the math here, though it worries me that it's not at all clear that a K80 will be much faster than a Titan X for a given problem. E.g. https://www.amax.com/blog/?p=907. It would also be really nice to get some pricing and benchmarks for the new TPUs, assuming they are priced better.

Maybe part of the problem is that vendors are not making it remotely easy to even understand what performance you'll actually get for a given price.

Re: Which GPU(s) to Get for Deep Learning

#86

Post by chronic940 » Tue, May 23, 2017, 5:06 AM UTC

zeptomu wrote: post #40

nightski wrote: post #15

Earlier quoted context omitted.

Not to mention, I can imagine it's frustrating uploading many gigs of data to amazon.

It is not that bad. Either you've your training data local (unlikely) or it is already available in the "cloud" (i.e. a public-facing service). Let's say a typical training set (raw data) is 100G. Assuming 2.5MB/s Upload capacity with local data, you've uploaded it to your deep learning machine in half a day (100000 / 2.5 / 3600 ~ 11 hours) - which is not that much, as most of your time will be used for development a…

You forgot the AWS Internet egress charge per GB..

Re: Which GPU(s) to Get for Deep Learning

#87

Post by shaklee3 » Tue, May 23, 2017, 5:27 AM UTC

tanilama wrote: post #78

boulos wrote: post #73

Earlier quoted context omitted.

Can you explain your math? A K80 on GCE is $.7/hr x 730 => $511/month if you were really 24x7. A K80 (and really we sell them by the die not the board) is more than $1000. I don't disagree that a consumer board is about that price, but they're not apples to apples. (Either in memory size, reliability or both). I'm fine with that being the real complaint: (major) cloud providers only sell the Tesla class boards, and t…

A GTX 1080 is like $550 at this point, K80 cannot really compete with the cost effectiveness here. And it is probably not going out of fashion in one year, so the price is totally worth it. The real attractiveness for cloud at this point is if you are going to train your model with 8-GPU or more, that is likely not feasible for individual enthusiasts, but demand for such machine is rare for hobbyists anyway.

This is kind of the key. If P100s were common on gce, it would be a better comparison. But the k80 is very old by today's standards, and a consumer card will run circles around it.

Re: Which GPU(s) to Get for Deep Learning

#88

Post by shaklee3 » Tue, May 23, 2017, 5:30 AM UTC

boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

One problem with this argument is there isn't enough history. When aws added k80 the price of the old ones did not decrease much; they just increased the price of the new k80 higher than the k20. Theoretically what you said can happen, but Google still has to run the k80s and pay for them.

By the way, I appreciate the useful comments you've left in the past on your experience for gce.

Re: Which GPU(s) to Get for Deep Learning

#89

Post by shaklee3 » Tue, May 23, 2017, 5:39 AM UTC

ktta wrote: post #83

neurotech1 wrote: post #39

Earlier quoted context omitted.

AFAIK AMD has not announced any Tensor Processing Unit or specific tensor acceleration capability. nVidia Volta [0] does have a Tensor Processing core optimization. https://www.nvidia.com/en-us/data-center/volta-gpu-architect...

'Tensor processing' isn't some special kind of compute. The TPUs (which were badly named by google) are just specifically designed ASICs with lower precision which have much lower power consumption than GPUs, therefore cost effective for inference, which google does more than anyone on earth. 'Tensor Processing Unit' has become some what a phrase used to confuse (trick?) people into thinking it's some new type of pro…

That's not true. Previous gpus, like Pascal, have lower precision instructions for fp16 and int8. The tpu allows a 4x4 matrix multiply and accumulated in one clock cycle through special cores. They're physically different parts of the die.

Re: Which GPU(s) to Get for Deep Learning

#90

Post by a_d » Tue, May 23, 2017, 6:26 AM UTC

boulos wrote: post #61

Disclaimer: I work on Google Cloud. I saw a lot of "should we use Cloud, no its crazy a GPU only costs $X". The key is that if you believe GPUs are going to get updated every year, and/or the best thing for ML may change (see TPU and plenty of startups with custom hardware) then suddenly buying hardware for 24? 36? months isn't as obvious. We (and AWS and Microsoft) have K80s because Maxwell wasn't a sufficiently fri…

Another option is to consider something that is even more abstracted from Google Cloud, AWS or Microsoft - such as, https://www.floydhub.com/ (Heroku for Deep learning). Ultimately, someone like them may be quicker to switch between different providers than individual companies.

(I have no association with Floyd)