Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Scale Deep Learning – Jeff Dean [pdf]

static.googleusercontent.com

Post reply on HN
Print view View original

11–20 of 44 posts

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#11

Post by sanxiyn »

nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

> Page 60-70 talks about paragraph embedding. I haven't seen this published before.

Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#12

Post by nl »

sanxiyn wrote: post #11
nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

> Page 60-70 talks about paragraph embedding. I haven't seen this published before. Paragraph embedding was published this year as "Distributed Representations of Sentences and Documents". http://arxiv.org/abs/1405.4053

Thanks.

It's interesting how the state of the art is outpacing publishing.

From a quick scan that appears quite similar to the approach in papers like "Parsing Natural Scenes and Natural Language with Recursive Neural Networks" (2011)[1]. Edit: I see they cite this paper too.

[1] http://nlp.stanford.edu/pubs/SocherLinNgManning_ICML2011.pdf

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#13

Post by zonzo »

andrewcamel wrote: post #4
31reasons wrote: post #3

Earlier quoted context omitted.

Its not as complicated as it seems. 2 full-time year might be required if you want to start in the PhD program for it but not for its applications.

That's pretty comforting to hear. Do you know of any resources off the top of your head to learn this type of thing?

torch7 used by facebook and google. Fast becoming the standard industrial neural nets library. http://torch.ch Read the tutorials and you can get something working in a day

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#14

Post by Houshalter »

andrewcamel wrote: post #4
31reasons wrote: post #3

Earlier quoted context omitted.

Its not as complicated as it seems. 2 full-time year might be required if you want to start in the PhD program for it but not for its applications.

That's pretty comforting to hear. Do you know of any resources off the top of your head to learn this type of thing?

Geoffrey Hinton (a notable deep learning researcher) did a coursera course on neural networks awhile ago. It's over but you can still see the lectures which are very good: https://www.coursera.org/course/neuralnets

Metacademy is also a very useful resource for anything machine learning: http://www.metacademy.org/

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#15

Post by jberryman »

nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

> Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding.

Actually, I'm not sure we can do those things in quite 0.1 sec. All I know about this is from around minute 9 from http://www.radiolab.org/story/267176-never-quite-now/ . One guest on the show even estimates thinking the simplest thought to be on the order of 0.25-0.5 sec.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#16

Post by xfs »

nl wrote: post #8

This whole slide deck is worth reading. A couple of highlights: Pg 26, quote: "Anything humans can do in 0.1 sec, the right big 10-layer network can do too". That is a very bold claim. It encompasses the entire fields of image and voice recognition as well as knowledge encoding. It's slowly becoming clear that this is likely to be true. Pg 39, 40: Google's ImageNet-winning system in 2011 had 7 layers and an error rat…

"Anything humans can do in 0.1 sec, the right big 10-layer network can do too"

Actually this was argued by Connectionists in 1980s. It is called Feldman's 100-step rule:

The critical resource that is most obvious is time. Neurons whose basic computational speed is a few milliseconds must be made to account for complex behaviors which are carried out in a few hundred milliseconds (Posner, 1978). This means that entire complex behaviors are carried out in less than a hundred time steps. Current AI and simulation programs require millions of time steps.

Feldman, J. A., & Ballard, D. H. (1982). Connectionist models and their properties. Cognitive Science, 6, p. 206.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#17

Post by kastnerkyle »

brandonb wrote: post #9

For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus…

Bookwise, Yoshua Bengio, Aaron Courville, and Ian Goodfellow are nearly finished with their MIT Press book on deep learning: http://www.iro.umontreal.ca/~bengioy/dlbook/ . It is pretty strong on the true theory of what is going on in deep networks, and has fairly good intuition for how and why things work. Paired with the deep learning tutorials http://www.deeplearning.net/tutorial/, as well as the content from UFLDL it is a pretty strong foundation for advanced study.

Michael's book seems to target a more introductory level - a beginner might be better off to start with that, follow with Andrew Ng's ML course, which has a section on neural nets including an assignment implementing backpropagation, then continue with the deep learning book and the {deep learning, UFLDL} tutorials. This should be solid enough to at least read most of the cutting edge work and papers, if that is the aim.

Hugo Larochelle's youtube course https://www.youtube.com/playlist?list=PL6Xpj9I5qXYEcOhn7Tqgh... and Hinton's coursera course https://www.coursera.org/course/neuralnets are also great references.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#18

Post by sytelus »

andrewcamel wrote: post #2

Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…

Here's another way to see: What have you been doing for past two years? Now imagine you had started learning this about 2 years ago. You would have been done by now and ready to tackle some of the most interesting problems instead of continue to do same boring stuff you had been doing for past 2 years. In 2016, come back here and look at this comment again :).

PS: For people who are saying you can "apply" DNNs in a day or learn it by a coursera course in 6 weeks - they are only very superficially right. Yeah, anyone can build ML model for a sample training data using tool in the same sense that anyone can compile sample code and have a working app. The problem is that most models don't work the first time as expected. The challenge lies in debugging the model and fix many of N possibilities to make it work. This is what working in ML is all about. It's like usual programming where it takes years of experience to debug the code and make it work for your purpose. The added twist in ML is that debugging is almost entirely statistical. When your model doesn't work, it doesn't work only in statistical sense. Your problem would be essentially that the model doesn't give expected answer this 12% of the time. For this 12% of the time, it doesn't work not because of some wrong "if" condition or misplaced subroutine call. The debugging is almost always statistical debugging - there are no breakpoints to put or no watch to set or not even exceptions. So it takes pretty solid background in statistics and probability to effectively work in ML. And yes, most likely it would take much more than 2 years.

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#19

Post by kastnerkyle »

andrewcamel wrote: post #2

Reading through documents like this really pains me because I it seems like such interesting work and I immediately want to understand it better, but then realize the time required to acquire the knowledge and experience necessary to understand and apply this technology is so great, that it almost seems like a waste of time. After all, think of all the things one could build in the 2 full-time years it would take to…

Truly understanding this to the point where you are "caught up" with the field may take 2 years, but one of the big blessings of deep learning is abstraction. You can go from very high level "black box" approaches simply using and following example code from Torch, Theano, or Caffe, all the way to nitty gritty study of the details of various architectures, how to optimize them, and how to apply them.

Watching videos of presentations and reading slides is often much easier than comprehending papers, though ultimately the paper should have much richer detail.

Personal anecdote: Two years ago I just started learning about these things, coming from an undergraduate degree in electrical engineering. Now I am in graduate school for deep learning and AI working to push things forward, one small step at a time. It is totally possible to learn this stuff in a reasonable amount of study, and there are more free resources than ever. Note that I had a full time engineering job until 6 months ago... doing something totally different!

Re: Large Scale Deep Learning – Jeff Dean [pdf]

#20

Post by brandonb »

kastnerkyle wrote: post #17
brandonb wrote: post #9

For those of you who want to learn the nuts and bolts of deep neural networks, Andrew Ng's tutorial on Unsupervised Feature Learning and Deep Learning is getting older but still great: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial The best research results from 2014 and 2013 make less use of the unsupervised techniques than initially expected, so I would start by focusing on the below sections, which focus…

Bookwise, Yoshua Bengio, Aaron Courville, and Ian Goodfellow are nearly finished with their MIT Press book on deep learning: http://www.iro.umontreal.ca/~bengioy/dlbook/ . It is pretty strong on the true theory of what is going on in deep networks, and has fairly good intuition for how and why things work. Paired with the deep learning tutorials http://www.deeplearning.net/tutorial/ , as well as the content from UFLD…

Didn't realize Yoshua & co had a book coming! That would definitely be the one to read.

BTW, for anybody who wants to learn machine learning in general, Kyle's blog also seems to be packed full of clear explanations with working demo code: http://kastnerkyle.github.io/

Very nice!

Post reply on HN