Large Scale Deep Learning – Jeff Dean [pdf]
static.googleusercontent.com
Large Scale Deep Learning – Jeff Dean [pdf]
1–10 of 44 posts
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#2Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to fully comprehend all of this to the point where it's useful in any practical application.
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#3Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…
Its not as complicated as it seems. 2 full-time year might be required if you want to start in the PhD program for it but not for its applications.
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#4Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…
Its not as complicated as it seems. 2 full-time year might be required if you want to start in the PhD program for it but not for its applications.
That's pretty comforting to hear. Do you know of any resources off the top of your head to learn this type of thing?
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#5Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…
You should be complaining about the computing resources to train 24 hidden layers.
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#6The second half of this video has an earlier talk that goes along with the slides: http://youtu.be/S9twUcX1Zp0?t=22m49s
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#7Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…
You should be complaining about the computing resources to train 24 hidden layers.
GPUs make a lot of this tractable. Nvidia is actually offering some free compute time for researchers:
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#8This whole slide deck is worth reading. A couple of highlights:
Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true.
Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rate of around 16%. The 2014 system had 24 layers and an error rate of 6.66%. Note that trained humans have an error rate of around 5%[1].
Page 50-57 talk about the miracle that is Word2Vec, and what is possible with that.
Page 60-70 talks about paragraph embedding. I haven't seen this published before.
Page 70-73 extends word/paragraph embedding for translation. I've seen a slide deck showing this works before, but I need to read the new paper cited there.
Page 74+ talks about cross-modal embeddings, especially the caption generation stuff. HN has had a few things on that over the past month or so.
[1] http://karpathy.github.io/2014/09/02/what-i-learned-from-com...
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#9For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great:
http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial
The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus more on supervised learning with deep neural networks:
Sparse Autoencoder: Neural Networks, Backpropagation Algorithm
Building Deep Networks for Classification: Deep Networks: Overview, Fine-tuning Stacked AEs
Working with Large Images: Feature extraction using convolution
You'll need some background in matrix algebra, calculus, and probability to understand this. Having taken a previous machine learning course, although not strictly necessary, is probably extremely helpful--I'd recommend taking any standard course on ML on Coursera or Udacity, or going through any standard textbook.
EDIT: I almost forgot that Michael Nielsen (who wrote the standard textbook on quantum computation) is also writing a free online textbook on Neural Networks and Deep Learning. Chapters 1-4 are currently available and would get you pretty far: http://neuralnetworksanddeeplearning.com/
Re: Large Scale Deep Learning – Jeff Dean [pdf]
#10This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…
"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"
That's an interesting direction from which to view things.
Then AI progress can be measured by increasing that timeframe.
Though I suspect there are some pretty gigantic discontinuities in there. The things humans can do in 3-4 seconds are qualitatively different from what they can do in less than 1 second, for example.
Still, it's a useful perspective to keep in mind.