Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

41–50 of 148 posts

Re: Why deep learning works even though it shouldn’t

#41

Post by blackbear_ »

aborsy wrote: post #36
blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

This is just bunch of simulation, not a derivation.

True, but it has been confirmed experimentally several times. Unfortunately theoretical understanding in deep learning is quite lagging behind practical progress.

This paper [1] seems to be getting closer to what you are looking for.

[1] https://arxiv.org/abs/1910.02875

Re: Why deep learning works even though it shouldn’t

#42

Post by morei »

quotemstr wrote: post #3
yters wrote: post #2

How do we define 'works'?

Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?

I think voice recognition would be a better example. It went from 'toy' to 'everyday use' because the word error rate dropped an order of magnitude.

Re: Why deep learning works even though it shouldn’t

#44

Post by candiodari »

moultano wrote: post #12
jostmey wrote: post #7

The author argues that deep-learning has abandoned statistics. I could not disagree more! Too much of the field was concerned with detailed proofs and mathematical formalism that were somewhat disconnected from probability theory. Modern machine learning (or AI or whatever) still has strong roots in probability and statistics. Loss functions are still based on concepts such as the log-likelihood function. Formal proo…

That isn't what I'm arguing at all.

I think this is a way to really piss off statisticians:

  Stop talking about minima. Stop talking about how your optimization algorithm behaves around a minimum. Nobody ever trains their model remotely close to convergence. ADAM doesn’t even provably converge. All real models diverge! You are nowhere close to a minimum! Stop talking about minima already goddamnit! Why even think about minima?! Minima are a myth! Everybody proves their results for minima of a convex function.
Actually arguing for divergent models. I mean, I love it, just because it's so very very cruel.

Re: Why deep learning works even though it shouldn’t

#45

Post by moultano »

Schiphol wrote: post #43

I loved this post, thanks for writing it. I get the argument why one shouldn't expect local minima in very high dimensions. But then, what's wrong with the informal argument that there has to be a minimum because, well, the expected loss cannot be negative?

With squared loss where it's easy for the loss to be zero, then yes, it will have lots of global minima, all at a loss of zero. For losses that asympotote, like logloss, they may have no minima.

Re: Why deep learning works even though it shouldn’t

#46

Post by hammock »

blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

Explain in layman's terms?

Re: Why deep learning works even though it shouldn’t

#47

Post by yters »

morei wrote: post #42
quotemstr wrote: post #3

Earlier quoted context omitted.

Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?

I think voice recognition would be a better example. It went from 'toy' to 'everyday use' because the word error rate dropped an order of magnitude.

Is that due to ML or all the massive crowdsourcing and more keyword/search driven approach? e.g. has dictation apps like dragon improved substantially?

Re: Why deep learning works even though it shouldn’t

#48

Post by Schiphol »

moultano wrote: post #45
Schiphol wrote: post #43

I loved this post, thanks for writing it. I get the argument why one shouldn't expect local minima in very high dimensions. But then, what's wrong with the informal argument that there has to be a minimum because, well, the expected loss cannot be negative?

With squared loss where it's easy for the loss to be zero, then yes, it will have lots of global minima, all at a loss of zero. For losses that asympotote, like logloss, they may have no minima.

Thanks. I guess my worry was that, once you are doing extremely well and your loss is very low, gradients are no longer independent, and will tend to go mostly up. Is this wrong?

Re: Why deep learning works even though it shouldn’t

#49

Post by yters »

quotemstr wrote: post #3
yters wrote: post #2

How do we define 'works'?

Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?

i would say that's more a function of corpus than ML

plus is it really that much better? i still get weirdness out of google translate

Re: Why deep learning works even though it shouldn’t

#50

Post by joshribakoff »

Schiphol wrote: post #48
moultano wrote: post #45

Earlier quoted context omitted.

With squared loss where it's easy for the loss to be zero, then yes, it will have lots of global minima, all at a loss of zero. For losses that asympotote, like logloss, they may have no minima.

Thanks. I guess my worry was that, once you are doing extremely well and your loss is very low, gradients are no longer independent, and will tend to go mostly up. Is this wrong?

I’m an ML newb but I think this would be true only of a converged model. Your model could always technically diverge in another epoch if learning rate is high enough and you process a batch of extreme outliers

Even then, you may have still converged on a local optimum which was the take away I got from the article

Post reply on HN