Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's Wrong with Deep Learning?

docs.google.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: What's Wrong with Deep Learning?

#31

Post by rasz_pl »

discardorama wrote: post #22
cec wrote: post #20

Earlier quoted context omitted.

> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?

But not in the same brain. Sure, evolution happened and picked the right parameters; but today's brain comes with the hyperparameters baked in (with some small amount of randomness). It is able to do all the things it can do without the luxury of parallel training and tuning. For example: you don't need to show a baby a 1000 photos of mugs before it can tell what's a mug and what isn't. Just show it 1 example of a mu…

One data point is perfectly fine for NN, it doesnt stop learning after first sample

https://www.youtube.com/watch?v=1GhNXHCQGsM

Re: What's Wrong with Deep Learning?

#32

Post by sgt101 »

cec wrote: post #20
discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?

No, I don't think so. I think that evolution is a complex system in the sense that it creates feedback loops altering the fitness landscapes it optimizes over. Organisms interact with each other through competition and predation and with their environment, for example by liberating oxygen from water. If evolution were about optimization then the current set of organisms would be "fitter" compared with dinosaurs (for example). I don't fancy my chances vs. a T.Rex, more to the point I don't fancy anything that is around at the moment's chances v. a T.Rex!

Re: What's Wrong with Deep Learning?

#33

Post by kmicklas »

ma2rten wrote: post #25
discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

Lack of theory is actually mentioned as one of the issues in the presentations. I don't think your examples are good though, Max polling reduces noise. RuLU learn faster than Sigmoid or tanh.

> I don't think your examples are good though, Max polling reduces noise. RuLU learn faster than Sigmoid or tanh.

That's not theory, that's just observation of the results. Why should we expect it to work that way?

Re: What's Wrong with Deep Learning?

#34

Post by sgt101 »

The final 20 or so slides about building general AI using deep learning strike me as really interesting. Seymour Papet said that if you can fit concepts into your cognitive architecture then they are learned. I think that this part of the presentation speaks to a need to demonstrate this as "learning" proper. It's strange, because I believe that this needs to happen, but the idea that you would do it with an "all nn all the way down" architecture, rather than breaking out into a symbolic layer a-la SOAR just seems odd.

Re: What's Wrong with Deep Learning?

#35

Post by aswanson »

sgt101 wrote: post #32
cec wrote: post #20

Earlier quoted context omitted.

> I'm no biologist, but I don't think our brains are going around trying to do a grid search for the best hyperparameters. Is that not essentially the process of evolution through natural selection? One giant optimisation problem?

No, I don't think so. I think that evolution is a complex system in the sense that it creates feedback loops altering the fitness landscapes it optimizes over. Organisms interact with each other through competition and predation and with their environment, for example by liberating oxygen from water. If evolution were about optimization then the current set of organisms would be "fitter" compared with dinosaurs (for…

But the T.Rex was a local maxima. I don't fancy a T. Rexes chance against humankind with the technology, intelligence and group organization we have. The T.Rex was optimized for physical force, but that proved useless against asteroids. Humans would have a chance against that type of threat.

Re: What's Wrong with Deep Learning?

#36

Post by Houshalter »

discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location.

ReLUs fit combinations of piece-wise linear functions. Whereas sigmoids are more nonlinear and can be harder to optimize. They were originally continuous approximations of binary threshold functions.

All these things can approximate each other. Neurons can approximate the max function, and ReLUs can approximate sigmoids. So there really isn't much to fret over.

It's like asking for a theory of which programming language is better. In practice they will have different advantages in different domains, but they are all Turing complete.

Re: What's Wrong with Deep Learning?

#37

Post by joe_the_user »

Houshalter wrote: post #36
discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location. ReLUs fit combinations of piece-wise linear functions. Whereas sigmoids are more nonlinear and can be harder to optimize. They were originally continuous approximations of binary threshold functions. All these things can approximate each other. Neurons can approximate the max function, and ReLUs can ap…

It's like asking for a theory of which programming language is better.

There's nothing wrong with just ignoring programming-language theory and just deciding on one, seat of the pants style. But this is because programming as it exists now is a static "art form" with only marginal progress expected.

However, assuming deep learning currently works unexplainably well and one aims to scientifically explain that good working, one would want an explanation which guides one's approach to extending the process.

I've done a bit of applied math, where knowing which kind of function to pull out of one's toolbox for which situation was the really-smart-people's purview, a fairly well guarded folk-knowledge, actually. I'm used to the "little bit of this, little bit of that" kind of explanation for which functions to use when and why. If one weighs them long enough, I assume one can intuitively figure out what to do.

But if we're aiming to advance fundamentally beyond the state-of-the-art, we would aim to quantify these advantages and disadvantages, to automate one more layer. So here we really should know and have a "real" theory here.

Re: What's Wrong with Deep Learning?

#39

Post by nightski »

joe_the_user wrote: post #37
Houshalter wrote: post #36

Earlier quoted context omitted.

Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location. ReLUs fit combinations of piece-wise linear functions. Whereas sigmoids are more nonlinear and can be harder to optimize. They were originally continuous approximations of binary threshold functions. All these things can approximate each other. Neurons can approximate the max function, and ReLUs can ap…

It's like asking for a theory of which programming language is better. There's nothing wrong with just ignoring programming-language theory and just deciding on one, seat of the pants style. But this is because programming as it exists now is a static "art form" with only marginal progress expected. However, assuming deep learning currently works unexplainably well and one aims to scientifically explain that good wor…

Do you think no one is trying? Should researchers just ignore all results until the underlying theory is found? What if we don't find it for another 50 years? I find it incredibly hard to be critical in this situation.

Re: What's Wrong with Deep Learning?

#40

Post by spin »

fizixer wrote: post #5
jhartmann wrote: post #4

It is always impressive to me how both Professor LeCun, Bengio & Hinton stuck to their guns and worked on these problems while others were not so interested. I'm very glad Hinton's group at DNN Research was able to blow away the competition in the Imagenet challenge. Now many more people are working on these ideas and some really amazing things have been accomplished in a very short time. I can't wait to see where we…

Do you have any information about what were the hot topics just before this new resurgence of NNs (meaning around late 90's, and early 2000's). I know symbolic AI was big in 60s, and 80s, but not sure about recent past.

I really like the book "Pattern Recognition and Machine Learning" by Christopher Bishop. It's packed full of all the latest-and-greatest algorithms.

WRT your question, an interesting "feature" of that book is that it was published just before deep neural networks started taking off, so there's no mention of DNNs in the book. You can see what the world was like right when they started taking off.

Post reply on HN