See slides 134-135. It's amazing that works. They get induction without any understanding at all. It still gets the right answer.
Intelligence may be dumber than we thought it was.
What's Wrong with Deep Learning?
21–30 of 48 posts
Re: What's Wrong with Deep Learning?
#22If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…
> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?
But not in the same brain. Sure, evolution happened and picked the right parameters; but today's brain comes with the hyperparameters baked in (with some small amount of randomness). It is able to do all the things it can do without the luxury of parallel training and tuning.
For example: you don't need to show a baby a 1000 photos of mugs before it can tell what's a mug and what isn't. Just show it 1 example of a mug a couple of times, and from then on it's able to identify mugs and mug-like objects pretty reliably.
Re: What's Wrong with Deep Learning?
#23I would like to see or hear more regarding the theory slides - in particular on the objective being a piecewise polynomial, and the distribution of weights using random matrix theory. Anyone know where I could find more?
http://arxiv.org/abs/1412.0233 (and the references within he cites by Gerard Ben Arous)
[deleted]
Re: What's Wrong with Deep Learning?
#24Earlier quoted context omitted.
> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?
But not in the same brain. Sure, evolution happened and picked the right parameters; but today's brain comes with the hyperparameters baked in (with some small amount of randomness). It is able to do all the things it can do without the luxury of parallel training and tuning. For example: you don't need to show a baby a 1000 photos of mugs before it can tell what's a mug and what isn't. Just show it 1 example of a mu…
[deleted]
Re: What's Wrong with Deep Learning?
#25If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…
Lack of theory is actually mentioned as one of the issues in the presentations.
I don't think your examples are good though, Max polling reduces noise. RuLU learn faster than Sigmoid or tanh.
Re: What's Wrong with Deep Learning?
#26Earlier quoted context omitted.
> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?
But not in the same brain. Sure, evolution happened and picked the right parameters; but today's brain comes with the hyperparameters baked in (with some small amount of randomness). It is able to do all the things it can do without the luxury of parallel training and tuning. For example: you don't need to show a baby a 1000 photos of mugs before it can tell what's a mug and what isn't. Just show it 1 example of a mu…
> [the brain] is able to do all the things it can do without the luxury of parallel training and tuning.
I beg to differ. Newborn babies can hardly do anything. Their brains are undergoing "parallel training and tuning" 24/7 starting even before they are born. Babies train themselves on thousands of hours of visual stimuli to gain the 3D object recognition capabilities to reliably identify objects such as mugs.
Re: What's Wrong with Deep Learning?
#27It is always impressive to me how both Professor LeCun, Bengio & Hinton stuck to their guns and worked on these problems while others were not so interested. I'm very glad Hinton's group at DNN Research was able to blow away the competition in the Imagenet challenge. Now many more people are working on these ideas and some really amazing things have been accomplished in a very short time. I can't wait to see where we…
Do you have any information about what were the hot topics just before this new resurgence of NNs (meaning around late 90's, and early 2000's). I know symbolic AI was big in 60s, and 80s, but not sure about recent past.
In the case of vision, there were a lot of things going on, but support vector machines, sliding window search, descriptors with invariances to different transforms like SIFT and HOG, spatial pyramids, deformable parts models, and mixtures of gaussians are all hot topics that you'll regularly see in papers from the early 2000's. There was a lot of work on improving runtime for these techniques going on, since training SVM experts and evaluating anything over a sliding window search space is very expensive. You'll also see a lot of work on approaches rooted in graph theory like conditional random fields, different flavors of Markov models, min-cut/max-flow based algorithms, and other types of probabilistic graphical models. Most of this stuff is still in widespread use; AI is a big field.
Re: What's Wrong with Deep Learning?
#28It is always impressive to me how both Professor LeCun, Bengio & Hinton stuck to their guns and worked on these problems while others were not so interested. I'm very glad Hinton's group at DNN Research was able to blow away the competition in the Imagenet challenge. Now many more people are working on these ideas and some really amazing things have been accomplished in a very short time. I can't wait to see where we…
Do you have any information about what were the hot topics just before this new resurgence of NNs (meaning around late 90's, and early 2000's). I know symbolic AI was big in 60s, and 80s, but not sure about recent past.
[deleted]
Re: What's Wrong with Deep Learning?
#29If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…
I agree re: lack of theory. No easy answer on that one other than "keep looking and get more people to help". We are making major practical gains along the way (although many are quick to discount those--"that's it???"). It's science in practice, theory follows. I disagree re: the 1000s of titans thing. Google, Baidu, etc are building large GPU clusters and have basically shown "similar resources = similar results",…
Funny what you say about Schmidhuber. I took Hinton's machine learning class, and he spend more than half of it teaching about things that didnt work.
Re: What's Wrong with Deep Learning?
#30Earlier quoted context omitted.
Probabilistic models. Recent research often focuses on Bayesian models. Probabilistic models have never really gone away. This presentation by LeCun actually suggests embedding neural networks inside of various types of probabilistic models: factor graphs and conditional random fields. This is, for example, how speech recognition works: the output of a neural network is fed into a probabilistic model (a hidden Markov…
Actually, state of the art speech recognition has switched over to having a Recursive Neural Network directly run over the audio input. Take a look at the paper at http://arxiv.org/abs/1412.5567 and http://usa.baidu.com/deep-speech-lessons-from-deep-learning/ However combining learning features with other systems is a very powerful approach and combining SVM's on top of the learned features of a Neural Network I woul…
state of the art is still very much using WFSTs and DNNHMMs. IBM and Google are still beating baidu.