Is there a volume summary of research papers or recommended book(s) for DNN?
What's Wrong with Deep Learning?
11–20 of 48 posts
Re: What's Wrong with Deep Learning?
#12Earlier quoted context omitted.
Probabilistic models. Recent research often focuses on Bayesian models. Probabilistic models have never really gone away. This presentation by LeCun actually suggests embedding neural networks inside of various types of probabilistic models: factor graphs and conditional random fields. This is, for example, how speech recognition works: the output of a neural network is fed into a probabilistic model (a hidden Markov…
Actually, state of the art speech recognition has switched over to having a Recursive Neural Network directly run over the audio input. Take a look at the paper at http://arxiv.org/abs/1412.5567 and http://usa.baidu.com/deep-speech-lessons-from-deep-learning/ However combining learning features with other systems is a very powerful approach and combining SVM's on top of the learned features of a Neural Network I woul…
Not to nitpick. I just want people to realize there are actually recursive nets that rely on a parser to be built (this is the recursive net that relies on backpropagation through structure). Then there is the recurrent net (LSTMs,multimodal) that rely on backpropagation through time.
Talking to some of the users of Recursive nets, they will be renaming them to tree rnns which should help clear up confusion a bit.
Re: What's Wrong with Deep Learning?
#13Is that document available in some standard format? The player that's playing it from Google Docs is buggy, and about 20% of the slides display an error message.
"10:01:47.662 Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://drive.google.com/viewerng/img?id=ACFrOgBySwSrGvI-XLL.... (Reason: CORS header 'Access-Control-Allow-Origin' missing).1 "
Re: What's Wrong with Deep Learning?
#14Is that document available in some standard format? The player that's playing it from Google Docs is buggy, and about 20% of the slides display an error message. "10:01:47.662 Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at https://drive.google.com/viewerng/img?id=ACFrOgBySwSrGvI-XLL... . (Reason: CORS header 'Access-Control-Allow-Origin' missing).1 "
Look to the nav bar at the top of the page. The source file is (purportedly) a PDF and there's a download button.
Re: What's Wrong with Deep Learning?
#15I would like to see or hear more regarding the theory slides - in particular on the objective being a piecewise polynomial, and the distribution of weights using random matrix theory. Anyone know where I could find more?
http://arxiv.org/abs/1412.0233 (and the references within he cites by Gerard Ben Arous)
Re: What's Wrong with Deep Learning?
#16It is always impressive to me how both Professor LeCun, Bengio & Hinton stuck to their guns and worked on these problems while others were not so interested. I'm very glad Hinton's group at DNN Research was able to blow away the competition in the Imagenet challenge. Now many more people are working on these ideas and some really amazing things have been accomplished in a very short time. I can't wait to see where we…
Do you have any information about what were the hot topics just before this new resurgence of NNs (meaning around late 90's, and early 2000's). I know symbolic AI was big in 60s, and 80s, but not sure about recent past.
kernels
Re: What's Wrong with Deep Learning?
#17It is always impressive to me how both Professor LeCun, Bengio & Hinton stuck to their guns and worked on these problems while others were not so interested. I'm very glad Hinton's group at DNN Research was able to blow away the competition in the Imagenet challenge. Now many more people are working on these ideas and some really amazing things have been accomplished in a very short time. I can't wait to see where we…
Do you have any information about what were the hot topics just before this new resurgence of NNs (meaning around late 90's, and early 2000's). I know symbolic AI was big in 60s, and 80s, but not sure about recent past.
As far as I remember, everyone was jumping on top of SVMs and Markov Models.
Re: What's Wrong with Deep Learning?
#18If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other?
I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Most DL results today are the result of throwing 1000s of Titans on the problem and then sitting back for a week for the beast to cough up a solution.
Tangential nitpick: one (very minor) nit I have with Prof LeCun's presentations is that I don't see him give more credit to Hinton and Schmidhuber. Hinton is mentioned a couple of times (3), but Schmidhuber is totally ignored; for example, when he mentions LSTM, it's cited as [Hochreiter 1997], even though it was a join publication with Schmidhuber. It should be cited as [Hochreiter et 1997], as he does in the very next line.
Re: What's Wrong with Deep Learning?
#19If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…
I agree re: lack of theory. No easy answer on that one other than "keep looking and get more people to help". We are making major practical gains along the way (although many are quick to discount those--"that's it???"). It's science in practice, theory follows.
I disagree re: the 1000s of titans thing. Google, Baidu, etc are building large GPU clusters and have basically shown "similar resources = similar results", but everyone else is mostly using single machine--maybe multi-GPU--and doing fine. A single Titan X is a BEAST for deep learning--nobody is using 1000s and you only need 1 for great results on most datasets I've seen.
On the subject of Schmidhuber, I saw him speak once and he spent half the talk explaining how he invented everything he's talking about (EVERYTHING!) and the other half talking about how no one gives him credit. I'm half joking, but I think there's more to his story. Or it's a miscarriage of justice.
Re: What's Wrong with Deep Learning?
#20If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…
> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters.
Is that not essentially the process of evolution through natural selection? One giant optimisation problem?