Why deep learning works even though it shouldn’t
moultano.wordpress.com
Why deep learning works even though it shouldn’t
1–10 of 148 posts
Re: Why deep learning works even though it shouldn’t
#2How do we define 'works'?
Re: Why deep learning works even though it shouldn’t
#3How do we define 'works'?
Are you not familiar with how awful machine translation was ten years ago and how eerily good it is today?
Re: Why deep learning works even though it shouldn’t
#4> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all.
Can someone who knows more about DL than I do help me understand this a little better?
The article uses the analogy of walls:
> Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to be surrounded by walls. In 4 dimensions, you can walk through walls. GPT3 has 175 billion parameters. In 175 billion dimensions, walls are so far beneath your notice that if you observe them at all it is like God looking down upon individual protons.
I'm struggling to understand what this really means in 4+ dimensions. But when I try to envision it going from 1 or 2 to 3 dimensions, it doesn't seem obvious at all that a 3D space should have fewer local optima than a 2D space.
In fact, having a "universal function" like a deep network seems like it should have more local optima. What am I missing?
Re: Why deep learning works even though it shouldn’t
#5A lot of people talk about minima because thats the language we have for analyzing optimization techniques. Deep Learning is still new enough that there is lots of low hanging fruit to explore including empirical approaches and applying existing theoretical tools to try and explain DNNs. The community is slowly moving towards developing new tools specifically for deep learning properly analyze these networks and prove stuff (bounds, convergence etc) about them.
Re: Why deep learning works even though it shouldn’t
#6> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…
Think of the model as a point in high d space that you're trying to trap inside a cage. (Corresponding to it being at a local minima and surrounded by higher points.) So you're trying to build a cube in n dimensions, which has 2n sides. Now imagine that each of those sides will randomly be there or not, probabilistically, with probability p. To actually trap a model like gpt3, p^175,000,000,000 has to be high enough that you observe a case during training where you roll a 1.
Re: Why deep learning works even though it shouldn’t
#7The author argues that deep-learning has abandoned statistics. I could not disagree more! Too much of the field was concerned with detailed proofs and mathematical formalism that were somewhat disconnected from probability theory. Modern machine learning (or AI or whatever) still has strong roots in probability and statistics. Loss functions are still based on concepts such as the log-likelihood function.
Formal proofs and mathematics are essential, but can become a distraction from the end goal. It is like playing Chess by going after your opponents pawns instead of their king. I would say modern machine learning has become tantamount to experimental physics and this article is written from the perspective of a string theory theorist.
Re: Why deep learning works even though it shouldn’t
#8> High dimensional spaces are unlikely to have local optima, and probably don’t have any optima at all. Can someone who knows more about DL than I do help me understand this a little better? The article uses the analogy of walls: > Just recall what is necessary for a set of parameters to be at a optimum. All the gradients need to be zero, and the hessian needs to be positive semidefinite. In other words, you need to…
The “physicist” explanation that I heard (meaning non-rigorous but good for building intuition) is that at every point where the derivative vanishes, for suitably random functions, every direction you move in will either be a direction where you increase or decrease at about 50-50 odds. In D dimensions there are 2D independent directions to rise or fall in (e.g. in D=2 dimensions, there’s north south east west), so there’s about 1/2^(2D) odds that any given critical point is actually a minimum if those probabilities are independent. That gets small really fast at large D.
Obviously this is not rigorous, though.
Re: Why deep learning works even though it shouldn’t
#9Woah! talk about sh-tting on deep learning and knowing jack about double descent, deep double descent.
flag-worthy sh-tpost.
Re: Why deep learning works even though it shouldn’t
#10Certainly a much shorter way to say this: if you have enough lines you can approximate any curve within a margin. This is what large neural networks are doing.
Another way to look at it: most neural nets are just a bunch of polynomials stitched together. You can see this from the popularity of the relu activation function. when the relu is negative, that poly is always zero in that area. When positive it's some poly multiplied by a const - another poly.
For nets that use other activation fns, they try to be linear in the area of most active input. So again they approximate a const * a ploy.