I liked that part at the beginning where the author made clear they were going to discuss intuitions that, while they aren't proven, would be useful to make explicit for a more general audience. Good candor.
Why deep learning works even though it shouldn’t
21–30 of 148 posts
Re: Why deep learning works even though it shouldn’t
#22> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…
Consider trying classifying shapes into either square or circle. The model outputs probability of circle. The absolute best you can do is to completely learn the training set. Assign 1 to all the circles and 0 to all the squares.
It is typical to squish the set of all real numbers into the interval 0 to 1. Any finite value will be squished to a value less than 1. So the model tries to make the number go higher and higher. No matter which model you have, you can always go a bit higher. Thus there is no optimal model.
The author mentions regularization, but strangely he then proceeds as if it didn't exist. Because with regularization, you can prove that there is a minimum. Basically (don't worry if you don't understand, I just include for other readers): loss goes to infinity as parameter norm goes to infinity, loss has a lower bound so it has a highest lower bound. Take a sequence of points in parameter space with loss converging to this bound. By Bolzano Weirstass it has a convergent sub sequence. Loss is continuous function of parameters, so the loss of the limit point is the limit of the loss. I.e. it's a minimum.
Re: Why deep learning works even though it shouldn’t
#23> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…
My intuition here is that there is always a fair bit of noise in the (data -> label) pairs. Models tend to air on the side of being _too_ expressive to compensate (just add another layer right?). Assuming our model is too expressive, we must turn off training before we get too far down and start memorizing training datasets. Put another way, one input pair will want to go down one path, another will want to go down a totally different path. Assuming the model is too expressive, we actually don't want to reach a minima, because that essentially guarantees we've overfit.
What the author is saying is that very quickly optimization becomes a maze, and can quickly turn into a combinatorial game. Each input starts down its own set of corridors (parameters at certain values) and it can take an _extremely_ long time for this maze to end. Any noise in the (data, label) pairs can make this maze have no end at all if the model is too small. If the model is too big, it's a moot point because it will have overfit at this point.
Re: Why deep learning works even though it shouldn’t
#24Earlier quoted context omitted.
The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so t…
That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just because of high dimensionality?
It’s a heuristic argument that critical points are extremely unlikely to be local minima (ie positive definite second derivative). Loss surfaces of DNNs do typically have a global minimum (zero if they fit the training data exactly).
Re: Why deep learning works even though it shouldn’t
#25Earlier quoted context omitted.
That makes sense and explains why it's unlikely to have local minimas, but I don't get the "no minimal at all" argument. Why no global minimas at all, just because of high dimensionality?
> but I don't get the "no minimal at all" argument. That's just wrong, no need to think about it. Only unbounded loss functions don't have minima, but using such a loss function would not make sense.
This isn't true. A loss function that asymptotes can also have no minima, which commonly used loss functions do.
Re: Why deep learning works even though it shouldn’t
#26How do we define 'works'?
Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?
I'm definitely not familiar with that. From my perspective it was mediocre but workable 10 years ago, and it's marginally better now if the languages are closely related.
Re: Why deep learning works even though it shouldn’t
#27Hey @moultano in response to your argument about walls and Nets not being in a minima, its my understanding nets always live on high dimensional saddle points and that's commonly referred to in literature. Even when you're optimizing you're just moving towards ever lower cost saddles that are closer to the optimum but almost never a local optimum (for the reasons spelled out in your post).
Thank you. Several people have pointed that out, and I'm probably not reading the right papers. Is it common when people introduce a new flavor of adaptive SGD to address how it handles saddles specifically? It is probably just a a matter of what manages to bubble up to me rather than what work is actually getting done, but I felt like the non-convergence of ADAM got talked about a lot, but haven't seen people talking as much about how optimizers behave differently on the landscapes we actually observe.
Re: Why deep learning works even though it shouldn’t
#28Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…
You are missing the point. The surprising thing about deep learning is that it can generalize to unseen data so well. Polynomial regression cannot.
If there is a pattern in data, such as it fitting on a curve, and you approximate that curve, then that should generalize to unseen data. What's surprising about that? A single polynominal regression wouldn't be able to do it, because some curves cannot be expressed as a polynominal, but superposition of multiple polynominals is apparently good enough.
Re: Why deep learning works even though it shouldn’t
#29Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing. Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly…
> if you have enough lines you can approximate any curve within a margin
How avoid overfitting the training data?
Re: Why deep learning works even though it shouldn’t
#30Earlier quoted context omitted.
Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?
I'm definitely not familiar with that. From my perspective it was mediocre but workable 10 years ago, and it's marginally better now if the languages are closely related.
There exist more objective measurements of translation quality which if you take to be relevant, show that machine translation quality has gotten better between most languages. https://en.wikipedia.org/wiki/Evaluation_of_machine_translat...
Here's a blogpost by google showing scores creeping up since 2009 https://ai.googleblog.com/2020/06/recent-advances-in-google-...
Also I agree that I don't recall it being awful like the parent poster suggests but maybe only because I reserve 'awful' for the type of rule based machine translation used before statistical approaches came onto the scene.