Why deep learning works even though it shouldn’t • Hacker News Board

Re: Why deep learning works even though it shouldn’t

#61

Post by moultano » Tue, Oct 20, 2020, 9:19 PM UTC

an_d_rew wrote: post #19

zachf wrote: post #8

Earlier quoted context omitted.

The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so t…

But you have to be careful about that word "independent". There's a reason that things like 3D protein structure estimation, for example, are still very difficult problems, because none of the coordinates are even approximately independent of the others. So you're back to a standard "minimization is really difficult" even in ultra-high dimensional spaces.

Yeah I was thinking about that as I was writing and trying to convey why I feel like deep models are different. I think one way of thinking about it is that protein structure, even though it has lots of parameters, it is all happening within the confines of 3D space. A protein that could move in lots of dimensions at once, could probably reliably fold much more easily, and it would be easy to find this structure.

Re: Why deep learning works even though it shouldn’t

#62

Post by pundesigner » Tue, Oct 20, 2020, 9:19 PM UTC

hammock wrote: post #46

blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

Explain in layman's terms?

Deep learning is a form of optimization. Optimization involves moving along a high-dimensional surface to find the lowest point. In principle this can be nearly impossible because the surface might be covered in dramatic peaks/valleys/saddles obscuring the route to the lowest point. Some simulations have implied that this is not what the surfaces corresponding to deep networks look like, and that they instead look like a big gentle slope down to the minimum, with only small bumps along the way.

Re: Why deep learning works even though it shouldn’t

#63

Post by muppet_frog » Tue, Oct 20, 2020, 9:26 PM UTC

blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

...hmm, that was counter to my understanding (limited though it may be...) which was partially formed by this paper: https://arxiv.org/abs/1712.09913

TLDR - loss landscapes are nasty, but you can tame them with skip connections.

Re: Why deep learning works even though it shouldn’t

#64

Post by UncleOxidant » Tue, Oct 20, 2020, 9:27 PM UTC

blackbear_ wrote: post #31

There is another reason why training deep neural networks is not as difficult as it sounds: the landscape of the loss function seems to be made of broad "U"-shaped valleys that gently descend towards a small loss region. At initialization, the network is likely close to such valley, and once it gets there the rest of training is just a leisurely stroll. Formally, people have studied the spectrum of the Hessian and fo…

Doesn't the loss function landscape depend a lot on what you're trying to get the neural network to learn (what problem you're trying to solve)?

Re: Why deep learning works even though it shouldn’t

#65

Post by YeGoblynQueenne » Tue, Oct 20, 2020, 9:28 PM UTC

I find the article style unreadable. Could someone please say whether the author explains why deep learning shouldn't work?

There is a bit at the start about how people in statistical learning throw their hands up at deep learning etc, but none of that makes sense to me. Neural nets are an idea as old as AI - even older, in fact. The need for deeper networks was well understood by the 1980's. There are well known results about feedforward neural nets with arbitrary hideen units being universal function approximators. Why shouldn't deep learning work?

Re: Why deep learning works even though it shouldn’t

#66

Post by lallysingh » Tue, Oct 20, 2020, 9:28 PM UTC

blackbear_ wrote: post #20

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.

How's that different from regular interpolation?

Re: Why deep learning works even though it shouldn’t

#67

Post by muppet_frog » Tue, Oct 20, 2020, 9:29 PM UTC

acadien wrote: post #17

Hey @moultano in response to your argument about walls and Nets not being in a minima, its my understanding nets always live on high dimensional saddle points and that's commonly referred to in literature. Even when you're optimizing you're just moving towards ever lower cost saddles that are closer to the optimum but almost never a local optimum (for the reasons spelled out in your post).

This paper makes the points that it's the saddles and not local minima that are the problem: https://arxiv.org/abs/1406.2572 It was the basis for adding 'momentum' to optimizers - so that you could skate across the saddles.

Re: Why deep learning works even though it shouldn’t

#68

Post by TillE » Tue, Oct 20, 2020, 9:37 PM UTC

quotemstr wrote: post #3

yters wrote: post #2

How do we define 'works'?

Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?

Japanese and Chinese translation to English is still absolute garbage outside of some obviously targeted sites (Wikipedia, I believe). Translating a German news site is still riddled with key errors because it doesn't understand how words and phrases get used in different contexts.

It's improved a bit, but it's still miles and miles away from any half-decent human translation.

Re: Why deep learning works even though it shouldn’t

#69

Post by asdfman123 » Tue, Oct 20, 2020, 9:42 PM UTC

pundesigner wrote: post #62

hammock wrote: post #46

Earlier quoted context omitted.

Explain in layman's terms?

Deep learning is a form of optimization. Optimization involves moving along a high-dimensional surface to find the lowest point. In principle this can be nearly impossible because the surface might be covered in dramatic peaks/valleys/saddles obscuring the route to the lowest point. Some simulations have implied that this is not what the surfaces corresponding to deep networks look like, and that they instead look li…

Just like in life, the imagined obstacles are scarier than the real ones.

Re: Why deep learning works even though it shouldn’t

#70

Post by pmiller2 » Tue, Oct 20, 2020, 9:48 PM UTC

nemoniac wrote: post #33

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

Perhaps you mean: "With four parameters I can fit an elephant, and with five I can make him wiggle his trunk."? https://en.wikiquote.org/wiki/John_von_Neumann

Side note: You can fit an elephant with just 1 parameter. See https://twitter.com/exobenelson/status/1001473539789213697 and the paper linked therein.