Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why deep learning works even though it shouldn’t

moultano.wordpress.com

Post reply on HN
Print view View original

11–20 of 148 posts

Re: Why deep learning works even though it shouldn’t

#11

Post by ozborn »

quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

I think the point being made is that a deep network (GPT-3 was the example with 175B parameters) will (due to the virtue of its size) not have any 'local optima' in the sense that there is no traditional 'local' for these high dimensional places. This is because as the # of dimensions increases it is easier to move away from or towards both better or worse parameter sets. Thus optimization algorithms don't have to be concerned about being trapped in a 'local optima'. Also because there are many good parameter sets not all parameters even need be used to get good results, thus processes like distillation can work.

Re: Why deep learning works even though it shouldn’t

#12

Post by moultano »

jostmey wrote: post #7

The author argues that deep-learning has abandoned statistics. I could not disagree more! Too much of the field was concerned with detailed proofs and mathematical formalism that were somewhat disconnected from probability theory. Modern machine learning (or AI or whatever) still has strong roots in probability and statistics. Loss functions are still based on concepts such as the log-likelihood function. Formal proo…

That isn't what I'm arguing at all.

Re: Why deep learning works even though it shouldn’t

#13

Post by salawat »

quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

The concept of optima is heavily dependent upon constraints on dimensionality.

If you're surrounded by walls at a 3D coordinate (walls in this sense is usually something akin to an a priori constraint on step size, which itself is the upper and lower bounds of the imposed delta introduced to the current step to try to find a new direction to go in numerical gradient descent), but you can arbitrarily "jump" 4 dimensionally, and there exists a point in the fourth dimensional space where your 3 dimensional space is no longer constrained from moving in the lower 3 dimensions again, you've essentially "avoided" a local optima, because your optimization function can continue to shift to find something better.

At least this holds if we're talking gradient descent, where the definition of an optima for a function is a point wherein any numerical deviation within your error tolerance always ends up converging to the same point.

If you take that same technique and apply it to higher dimensional spaces, the more dimensions you have, the less likely (in theory) your model is from getting stuck.

I know for a fact this doesn't always hold though, as almost every GPT2/3 model I come across I can still manage to get it snarled in predictive loops, where it does nothing but suggest the same thing over and over and over and over and over and over again, indicating a locally maximal optima for the predictor.

One of my favorite way to trip them up is generally some variant of "I once heard a story from a man who heard it from a man, who heard it from a man,... usually it sets it up for the loop. Sometimes you need to massage it a bit, but it's generally pretty easy to lead the predictor into a loop.

If you really want to blow the theory there are no higher dimensionality optima though, just look at other people. If there were not higher dimensional optima, why do bad habits exist, and get converged on so readily that we actively have to discourage, label, or avoid them?

The fact is that for a general function simulator, the trick isn't not falling victim to higher dimensional optima, but learning to recognize what and when you can safely tolerate some, and when you can't, because you really can't avoid the damn things in a resource or physically constrained problem space.

Re: Why deep learning works even though it shouldn’t

#14

Post by maxlamb »

zachf wrote: post #8
quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so t…

That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just because of high dimensionality?

Re: Why deep learning works even though it shouldn’t

#15

Post by blackbear_ »

maxlamb wrote: post #14
zachf wrote: post #8

Earlier quoted context omitted.

The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so t…

That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just because of high dimensionality?

> but I don't get the "no minimal at all" argument.

That's just wrong, no need to think about it. Only unbounded loss functions don't have minima, but using such a loss function would not make sense.

Re: Why deep learning works even though it shouldn’t

#16

Post by dougabug »

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

ReLU networks are piecewise linear.

Re: Why deep learning works even though it shouldn’t

#17

Post by acadien »

Hey @moultano in response to your argument about walls and Nets not being in a minima, its my understanding nets always live on high dimensional saddle points and that's commonly referred to in literature. Even when you're optimizing you're just moving towards ever lower cost saddles that are closer to the optimum but almost never a local optimum (for the reasons spelled out in your post).

Re: Why deep learning works even though it shouldn’t

#18

Post by blackbear_ »

quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

Your intuition is off, but only slightly. As the dimensionality increases, the number of stationary points (zero gradient) also increases. But they become overwhelmingly likely to be saddle points rather than minima.

You can read more here [1].

[1] http://ganguli-gang.stanford.edu/pdf/14.SaddlePoint.NIPS.pdf

Re: Why deep learning works even though it shouldn’t

#19

Post by an_d_rew »

zachf wrote: post #8
quicklime wrote: post #4

> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…

The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so t…

But you have to be careful about that word "independent".

There's a reason that things like 3D protein structure estimation, for example, are still very difficult problems, because none of the coordinates are even approximately independent of the others.

So you're back to a standard "minimization is really difficult" even in ultra-high dimensional spaces.

Re: Why deep learning works even though it shouldn’t

#20

Post by blackbear_ »

RandyRanderson wrote: post #10

Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…

You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.
Post reply on HN