Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What's Wrong with Deep Learning?

docs.google.com

Post reply on HN
Print view View original

41–48 of 48 posts

Re: What's Wrong with Deep Learning?

#41

Post by discardorama »

Houshalter wrote: post #36
discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location. ReLUs fit combinations of piece-wise linear functions. Whereas sigmoids are more nonlinear and can be harder to optimize. They were originally continuous approximations of binary threshold functions. All these things can approximate each other. Neurons can approximate the max function, and ReLUs can ap…

> Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location.

From Geoff Hinton's AMA on Reddit: The pooling operation used in convolutional neural networks is a big mistake and the fact that it works so well is a disaster.

Re: What's Wrong with Deep Learning?

#42

Post by discardorama »

modeless wrote: post #26
discardorama wrote: post #22

Earlier quoted context omitted.

But not in the same brain. Sure, evolution happened and picked the right parameters; but today's brain comes with the hyperparameters baked in (with some small amount of randomness). It is able to do all the things it can do without the luxury of parallel training and tuning. For example: you don't need to show a baby a 1000 photos of mugs before it can tell what's a mug and what isn't. Just show it 1 example of a mu…

> [the brain] is able to do all the things it can do without the luxury of parallel training and tuning. I beg to differ. Newborn babies can hardly do anything. Their brains are undergoing "parallel training and tuning" 24/7 starting even before they are born. Babies train themselves on thousands of hours of visual stimuli to gain the 3D object recognition capabilities to reliably identify objects such as mugs.

Newborn babies can hardly do anything because their parameters have not been set; but their hyperparameters (things like, to use NN analogies: activation function, learning rate, etc.) are baked in.

Re: What's Wrong with Deep Learning?

#43

Post by modeless »

discardorama wrote: post #42
modeless wrote: post #26

Earlier quoted context omitted.

> [the brain] is able to do all the things it can do without the luxury of parallel training and tuning. I beg to differ. Newborn babies can hardly do anything. Their brains are undergoing "parallel training and tuning" 24/7 starting even before they are born. Babies train themselves on thousands of hours of visual stimuli to gain the 3D object recognition capabilities to reliably identify objects such as mugs.

Newborn babies can hardly do anything because their parameters have not been set; but their hyperparameters (things like, to use NN analogies: activation function, learning rate, etc.) are baked in.

Right. Evolution picked the hyperparameters, and experiential learning picks the parameters. (To first order; certainly there are some instinctual behaviors programmed by evolution and I wouldn't be surprised if some hyperparameters are influenced by experience as well.)

Re: What's Wrong with Deep Learning?

#44

Post by jokoon »

discardorama wrote: post #18

If someone were to ask me what's wrong with DL (and not that anyone would, since I'm an unknown), I'd say the lack of theory. Most DL results look very hacky to me. Someone says Max Pooling works; someone again comes along and says it's not necessary. Someone says sigmoid or tanh are the best activation functions; someone else says ReLUs are better. And so on. Why? Why is one better than the other? I'm no biologist,…

> I'd say the lack of theory.

Well that's what makes it research then. It's a new field, and obviously there needs to be more science and creative thinking involved into drawing theories about deep learning. The field seems already pretty hard for beginners, so of course there will be less scientists involved into making theories.

Re: What's Wrong with Deep Learning?

#45

Post by versteegen »

rasz_pl wrote: post #29
dave_sullivan wrote: post #19

Earlier quoted context omitted.

I agree re: lack of theory. No easy answer on that one other than "keep looking and get more people to help". We are making major practical gains along the way (although many are quick to discount those--"that's it???"). It's science in practice, theory follows. I disagree re: the 1000s of titans thing. Google, Baidu, etc are building large GPU clusters and have basically shown "similar resources = similar results",…

Funny what you say about Schmidhuber. I took Hinton's machine learning class, and he spend more than half of it teaching about things that didnt work.

OK, but what's the relevance to Schmidhuber?

Re: What's Wrong with Deep Learning?

#47

Post by Houshalter »

discardorama wrote: post #41
Houshalter wrote: post #36

Earlier quoted context omitted.

Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location. ReLUs fit combinations of piece-wise linear functions. Whereas sigmoids are more nonlinear and can be harder to optimize. They were originally continuous approximations of binary threshold functions. All these things can approximate each other. Neurons can approximate the max function, and ReLUs can ap…

> Max pooling tests if a feature occurs anywhere in a certain area, rather than being sensitive to the exact location. From Geoff Hinton's AMA on Reddit: The pooling operation used in convolutional neural networks is a big mistake and the fact that it works so well is a disaster.

Hinton doesn't like that pooling loses track of the exact locations where features are located, and just tests if a feature occurs in some area.

The basic effect of this is to decrease the resolution, so it's more tractable to operate on. Without pooling you are stuck with a huge resolution at each layer.

Re: What's Wrong with Deep Learning?

#48

Post by abecedarius »

kragen wrote: post #38

For some reason, most of these slides say, "¡Vaya! Hubo un problema para cargar la página." Is there a better URL, maybe with the PDF itself? ( https://doc-04-4c-docs.googleusercontent.com/docs/securesc/h... doesn't look like it's going to work reliably for other people.)

I downloaded the pdf; will mail you.
Post reply on HN