Looks good! What performance gain could be expected when using GPU vs CPU for Deep Learning? It'd be great to get some figures.
Depends, of course, on the architecture but you could be looking at a ~x50 speedup.
GPU-Accelerated Deep Learning Library in Python
21–30 of 51 posts
Re: GPU-Accelerated Deep Learning Library in Python
#22Hm. And here I was thinking I'd seen most of the accelerated deep learning implementations. Although this one seems a touch incomplete as far as features go. There are some other for those not aware: [1] [Python] pylearn2 by the Lisa Lab @ uMontreal. Probably one of the beefiest of the bunch. Has a pretty extensive feature set, but IMO isn't the simplest to use. [2] [Python] MORB by Sander Dielman. Just restricted bo…
Yes, Hebel doesn't have a ton of features and a kitchen-sink of different models yet, but I hope that's going to change. There are lots of things that are quite easy to implement in the current framework, such as:
- Neural net regression
- Autoencoders
- Restricted Boltzman machines
There's a lot of interest for convolutional networks and the best way to implement it will be to wrap Alex Krizhevsky's cuda-convnet, like DeepNet and PyLearn2 have, but this will require a bit more effort.
With respect to other deep learning packages, Hebel doesn't necessarily do everything differently, but depending on your needs it may be the best choice for a particular job.
PyLearn2 is big and monumental and although I haven't used it much personally, it seems to excellent. But as you mentioned, it's not necessarily easy to use and if you want to extend it, you have to learn the Theano development model, which takes some time to grok.
DeepNet is quite similar to Hebel in its approach (even though it offers more models right now). However, DeepNet is based on cudamat and gnumpy, which I have found to often be quite unstable and slow. Hebel is based on PyCUDA which is very stable and according to some preliminary tests I did runs about twice as fast as cudamat.
So, the idea of Hebel is that it should make it easy to train the most important deep learning models without much setup or having to write much code. It is also supposed to make it easy to implement new models through a modular design that lets you subclass existing layers or models to implement variations of them.
Re: GPU-Accelerated Deep Learning Library in Python
#23All of my GPU resources are OpenCL based. The only place I could run it is on AWS. Someone really needs to make a CUDA compiler with an OpenCL backend.
Yes, I'm sorry about that, but it's not going to change. I personally don't have experience developing for OpenCL and the deep learning community seems to very much have embraced CUDA.
Re: GPU-Accelerated Deep Learning Library in Python
#24Looks a lot like Theano [ http://deeplearning.net/software/theano/index.html ]
Theano is for compiling mathematical expressions to CUDA, it doesn't implement any deep learning itself. If anything this may be comparable to pylearn2.
Yes, this is true. Theano is just a GPU-computing framework that you can use to implement any kind of numerical model. Hebel implements some specific models, but it uses PyCUDA as a backend.
Re: GPU-Accelerated Deep Learning Library in Python
#25Hm. And here I was thinking I'd seen most of the accelerated deep learning implementations. Although this one seems a touch incomplete as far as features go. There are some other for those not aware: [1] [Python] pylearn2 by the Lisa Lab @ uMontreal. Probably one of the beefiest of the bunch. Has a pretty extensive feature set, but IMO isn't the simplest to use. [2] [Python] MORB by Sander Dielman. Just restricted bo…
Yes, Hebel doesn't have a ton of features and a kitchen-sink of different models yet, but I hope that's going to change. There are lots of things that are quite easy to implement in the current framework, such as: - Neural net regression - Autoencoders - Restricted Boltzman machines There's a lot of interest for convolutional networks and the best way to implement it will be to wrap Alex Krizhevsky's cuda-convnet, li…
Question: do you/will you plan to support converting GPU nets to CPU, perhaps by keeping weights and architecture definition separate from PyCUDA dependent structures during serialization?
I have found that using a trained net for preprocessing can be accomplished using very limited resources (read: Core 2 Duo laptop). This is one of the very nice features of DeCAF, which could allow for some interesting applications on embedded devices.
Great work by the way - I look forward to testing it out soon!
Re: GPU-Accelerated Deep Learning Library in Python
#26Looks good! What performance gain could be expected when using GPU vs CPU for Deep Learning? It'd be great to get some figures.
Depends, of course, on the architecture but you could be looking at a ~x50 speedup.
Yes, you can't come up with a hard number, but a 10x speedup is really the baseline. 50x and more are not out of the question.
Re: GPU-Accelerated Deep Learning Library in Python
#27Earlier quoted context omitted.
If I had a 760 I'd likely just run all my experiments right on my device. As it stands my only machine is a Early 2009 macbook pro with a Geforce 9600M. The nice part about pylearn2, and theano is that the symbolic expressions compiled by theano can run on your processor too, albeit much slower. You can always test to make sure 1 to 2 epochs work locally before sending it off to AWS for GPU computation. I'd be very w…
The convolutional neural network code that pylearn2 and the Toronto group use is specifically tuned for GTX580 cards - users have reported factors of 2x-10x slowdown using Kepler series cards. In general, most users (of pylearn2 at least) highly recommend a GTX5xx device. I personally use a GTX570, and it is pretty decent, though not spectacular. Costwise, it is reasonably priced, and "good enough" for most of the ne…
Huh. Any theories as to why that is? Highly tuned coalesced reads that backfire on the kepler arch?
Re: GPU-Accelerated Deep Learning Library in Python
#28Earlier quoted context omitted.
If I had a 760 I'd likely just run all my experiments right on my device. As it stands my only machine is a Early 2009 macbook pro with a Geforce 9600M. The nice part about pylearn2, and theano is that the symbolic expressions compiled by theano can run on your processor too, albeit much slower. You can always test to make sure 1 to 2 epochs work locally before sending it off to AWS for GPU computation. I'd be very w…
The convolutional neural network code that pylearn2 and the Toronto group use is specifically tuned for GTX580 cards - users have reported factors of 2x-10x slowdown using Kepler series cards. In general, most users (of pylearn2 at least) highly recommend a GTX5xx device. I personally use a GTX570, and it is pretty decent, though not spectacular. Costwise, it is reasonably priced, and "good enough" for most of the ne…
I didn't realize the performance dip was so dramatic. As of late I've been considering acquiring some hardware and I guess I'm going to have to keep this in mind. If ones were to buy one today I don't even think I can find stores that sell the GTX 580. Would probably need to search on ebay.
Thanks for the heads up friend!
Re: GPU-Accelerated Deep Learning Library in Python
#29Earlier quoted context omitted.
The convolutional neural network code that pylearn2 and the Toronto group use is specifically tuned for GTX580 cards - users have reported factors of 2x-10x slowdown using Kepler series cards. In general, most users (of pylearn2 at least) highly recommend a GTX5xx device. I personally use a GTX570, and it is pretty decent, though not spectacular. Costwise, it is reasonably priced, and "good enough" for most of the ne…
Huh. Any theories as to why that is? Highly tuned coalesced reads that backfire on the kepler arch?
From what I understand, it is due to the programming specifics of the training algorithms, primarily being focused on exploiting certain registers and architecture features specific to Fermi. The code actually got updated from GTX280 series to GTX580 series IIRC, so it is likely that it will be updated again at some point by a motivated researcher or group. I suspect there simply isn't a need to update right now for most labs (though I suspect TITAN / TITAN LE / TITAN II may change that). Also, Alex Krizhevsky now works for Google :), so someone else may need to do the updating.
http://www.wired.com/wiredenterprise/2013/03/google_hinton/
You can check out the code here - it is really good IMO.
Re: GPU-Accelerated Deep Learning Library in Python
#30Earlier quoted context omitted.
The convolutional neural network code that pylearn2 and the Toronto group use is specifically tuned for GTX580 cards - users have reported factors of 2x-10x slowdown using Kepler series cards. In general, most users (of pylearn2 at least) highly recommend a GTX5xx device. I personally use a GTX570, and it is pretty decent, though not spectacular. Costwise, it is reasonably priced, and "good enough" for most of the ne…
I didn't realize the performance dip was so dramatic. As of late I've been considering acquiring some hardware and I guess I'm going to have to keep this in mind. If ones were to buy one today I don't even think I can find stores that sell the GTX 580. Would probably need to search on ebay. Thanks for the heads up friend!
Check out this discussion - it may help you decide what card to get. There was also an email somewhere about how TITAN is currently not any faster than a 580, though no hard numbers.
https://groups.google.com/forum/#!topic/pylearn-dev/cODL9RXP...
Once again, my 570 is slower than a 580 (about 2x), but "good enough" for now.