Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Which GPU(s) to Get for Deep Learning

timdettmers.com

Post reply on HN
Print view View original

111–119 of 119 posts

Re: Which GPU(s) to Get for Deep Learning

#111

Post by marmaduke »

technics256 wrote: post #103
marmaduke wrote: post #100

Earlier quoted context omitted.

> but they're not apples to apples It would be nice to see Nvidia or someone expand on this, so that users who have to make this choice can do so without guessing. If Google or AWS or M$ could publish reliability information, that'd be cool too. Illustrative case: I run Monte Carlo work on GPUs and administer a local compute cluster. I tested a workload on a 16 GB P100 and a GTX 1080. A 12 GB P100 costs (academic, EU…

Thank you for the answer. When you say performance difference is 2x, I presume the P100 is 2x faster than the 1080 in training epoch time?

yep

Re: Which GPU(s) to Get for Deep Learning

#112

Post by ktta »

shaklee3 wrote: post #89
ktta wrote: post #83

Earlier quoted context omitted.

'Tensor processing' isn't some special kind of compute. The TPUs (which were badly named by google) are just specifically designed ASICs with lower precision which have much lower power consumption than GPUs, therefore cost effective for inference, which google does more than anyone on earth. 'Tensor Processing Unit' has become some what a phrase used to confuse (trick?) people into thinking it's some new type of pro…

That's not true. Previous gpus, like Pascal, have lower precision instructions​ for fp16 and int8. The tpu allows a 4x4 matrix multiply and accumulated in one clock cycle through special cores. They're physically different parts of the die.

But they're not a new paradigm as they are often stated in 'marketing talk'. Also, what are you referring to by 'tpu'? Google's TPU or nVidia's Tensor processing core? Because AFAIK, there aren't any details about TPU's processing pipeline other than that one paper about data center use.

I realize they are physical parts of the die, which is why I said in my last sentence about how they take up unnecessary die space. Why? Because that die space will be better used for higher precision FP units which will be useful in training, which is more important than inference for most of the people in this thread.

Re: Which GPU(s) to Get for Deep Learning

#113

Post by solomatov »

ktta wrote: post #81
solomatov wrote: post #52

Earlier quoted context omitted.

>If that GPU is a real bottleneck for you, then you're much better off spending money on GCP/AWS's GPU offerings. That's because consumer GPUs get superseeded every year and online offering's price will only go down. GCP and AWS have old GPUs and they are really really expensive. If you expect to run workloads for a long time, it would be more cost efficient to buy your own hardware.

They are very, very few cases where getting an expensive GPU - anything more than a Nvidia 1060 would make financial sense. The 1050 is a beginner card is it is perfectly fine to learn and run small nets. More importantly, you can decide if machine learning is for you. Then, comes to second investment, which is actually running real-world models. Although the online GPU offering is expensive (you can also look around…

There are two problems with your approach:

* Setting up machine on AWS is more complicated than locally, and requires some admin skills.

* If you use spot instances, you need to handle checkpointing, which requires persistent storage, and all of this stuff requires even more admin skills

The goal of a person who starts working with deep learning is to learn deep learning, not how to setup machines, manage them, work with checkpoints, etc.

Also, don't forget that there's a large market for used GPUs and you can get real bargains.

Re: Which GPU(s) to Get for Deep Learning

#114

Post by varelse »

bmiranda wrote: post #69
varelse wrote: post #32

Earlier quoted context omitted.

There's a real opportunity for AMD here if NVIDIA doesn't release a consumer variant of Volta because Titan X Pascal intentionally cripples FP16 performance to drive P100 adoption. It's the same sort of nonsense they pulled with crippling FP64 performance on GTX Titan (Maxwell) to drive K20/K40/K80 adoption, but it got old a long time ago. What's more interesting is the disruptive change to the underlying programming…

Not having fast FP64 is a design decision. Space isn't free, and FP64 takes up a lot of space that could be used on delivering higher FP32 perf, which is more important for video games.

Except that's not how it played out. The Tesla GPU all the way through Maxwell was just a different binning of the GeForce GPU, usually clocked lower for passive cooling and improved stability (sure, K80 is an exception, but K80 is a strange and tragic evolutionary dead-end IMO).

Tesla P100 was the first real HW-level divergence with its 2x FP16 support. But because we still can't have nice things, GTX 1080 was the first GPU with fast INT8/INT16 instructions, followed by the mostly identical except much more expensive Tesla P40. So we ended up with the marchitecture nonsense that P100 was for training as P40 is for inference despite being mostly identical except as noted above.

I'll assume Volta unifies INT8/INT16/FP16? And I think it's OK if the Tesla card has higher tensor core performance, but if the tensor core on GeForce is slower than its native FP16 support, I can only conclude NVIDIA now hates its own developers and has decided to sniff its own exhaust pipe. Isn't having to refactor all existing warp-level code for thread-within-thread enough complication for one GPU generation?

Also, if consumer Volta ends up with craptastic FP16 support (ala 1/64 perf in GP102 vs GP100, slower than emulating it with FP16 loads and FP32 math), NVIDIA will create a genuine opening for AMD to be the other GPU provider in deep learning.

Re: Which GPU(s) to Get for Deep Learning

#115

Post by huac »

marvy wrote: post #59
KerrickStaley wrote: post #42

Question: if I'm learning about neural networks and want to e.g. train a network to recognize MNIST digits, do I need a discrete graphics card (probably attached to a VPS that I would rent)? Or can I use the i5 Kaby Lake (which has an integrated GPU) in my laptop to train my network?

As other people have said, for MNIST, CPU is fine. One core. You can get over 90% accuracy with a network with just 1 hidden layer with only a few seconds of training. Or something like that. If you've never played with MNIST before, it's kind of amazing how easy it is. For instance, the following idea "works", in that you get results that are pretty bad, but much better than chance. (Even more than 50% right, I thin…

A 0 hidden layer NN is just a linear (or logistic) regression.

On MNIST I think you get something like 60% accuracy.

Re: Which GPU(s) to Get for Deep Learning

#116

Post by shaklee3 »

ktta wrote: post #112
shaklee3 wrote: post #89

Earlier quoted context omitted.

That's not true. Previous gpus, like Pascal, have lower precision instructions​ for fp16 and int8. The tpu allows a 4x4 matrix multiply and accumulated in one clock cycle through special cores. They're physically different parts of the die.

But they're not a new paradigm as they are often stated in 'marketing talk'. Also, what are you referring to by 'tpu'? Google's TPU or nVidia's Tensor processing core? Because AFAIK, there aren't any details about TPU's processing pipeline other than that one paper about data center use. I realize they are physical parts of the die, which is why I said in my last sentence about how they take up unnecessary die space.…

Tpu is what Nvidia and Google both call their low precision matrix multiply hardware. Your assessment of how the hardware can better be used is just your assessment. The reason they did this is because they realized that a large share of their professional users want this, and they're willing to pay. They'll likely have other versions of the card where the tpus aren't there, so you'll have that option as well. It's not a new paradigm, but it's absolutely new to have this type of computation in a single cycle at these clock rates.

Re: Which GPU(s) to Get for Deep Learning

#117

Post by visionscaper »

zeptomu wrote: post #27

There is an interesting offer by Hetzner that's available for some month now. They provide a dedicated server with a GTX 1080 for ~99e/month (111$/month) with adequate CPU (i7-6700), 64G Memory, 500G disk space and 50TB Traffic - there are also on-demand offerings by GCP and AWS, but I do not think they can match the offer by Hetzner: https://www.hetzner.de/us/hosting/produkte_rootserver/ex51ss... . Keep in mind that…

I might be mistaken, but these servers just have one GTX 1080.

Re: Which GPU(s) to Get for Deep Learning

#118

Post by ktta »

solomatov wrote: post #113
ktta wrote: post #81

Earlier quoted context omitted.

They are very, very few cases where getting an expensive GPU - anything more than a Nvidia 1060 would make financial sense. The 1050 is a beginner card is it is perfectly fine to learn and run small nets. More importantly, you can decide if machine learning is for you. Then, comes to second investment, which is actually running real-world models. Although the online GPU offering is expensive (you can also look around…

There are two problems with your approach: * Setting up machine on AWS is more complicated than locally, and requires some admin skills. * If you use spot instances, you need to handle checkpointing, which requires persistent storage, and all of this stuff requires even more admin skills The goal of a person who starts working with deep learning is to learn deep learning, not how to setup machines, manage them, work…

>Setting up machine on AWS is more complicated than locally, and requires some admin skills.

There are tons of guides online where you can learn how to do so in >If you use spot instances, you need to handle checkpointing

Again, not a big deal to learn.

>The goal of a person who starts working with deep learning is to learn deep learning, not how to setup machines, manage them, work with checkpoints, etc.

I mean, if you're buying a GPU and setting it up, you're more than likely assembling the computer by yourself. You'll also have to maintain it properly. Then you'll have to look for correct drivers and other software which can get frustrating (it did for me).

On the other hand, I just could use a step by step process for the AWS instances since they had a few specific types of GPUs and I didn't have to even think, just copy/paste the commands from the webpage to the terminal. There are even AMIs which setup everything for you, which would require even less effort, but I don't trust them so I go with a clean disk.

Moreover, learning how to use AWS is a much more valuable skill than putting together computers so time well invested I would say.

Re: Which GPU(s) to Get for Deep Learning

#119

Post by marvy »

huac wrote: post #115
marvy wrote: post #59

Earlier quoted context omitted.

As other people have said, for MNIST, CPU is fine. One core. You can get over 90% accuracy with a network with just 1 hidden layer with only a few seconds of training. Or something like that. If you've never played with MNIST before, it's kind of amazing how easy it is. For instance, the following idea "works", in that you get results that are pretty bad, but much better than chance. (Even more than 50% right, I thin…

A 0 hidden layer NN is just a linear (or logistic) regression. On MNIST I think you get something like 60% accuracy.

Yes, it is just logistic regression. I just dug up my old code from a few weeks (months?) ago, and it seems that somewhere around 90% is not too hard. 60% is way too low, unless I'm measuring wrong somehow.
Post reply on HN