It's obvious no? For double descent:
the network with a billion parameters is so large that it learns to simulate a neural network itself which first uses the test data set to train itself and then produces the final overfitted values??
An overview of the theory of overparameterized machine learning
41–50 of 58 posts
Re: An overview of the theory of overparameterized machine learning
#42Earlier quoted context omitted.
> In practice, these huge models are, in laymans terms, fucking awesome and work really well A similarly surprising result from an adjacent community, Bayesian Statistics, is that in the case of hierarchical models, increasing your number of parameters can paradoxically reduce overfitting. The scale of parameters in Bayesian model's is no where near that of these deep neural nets, but nonetheless this is a similarly…
This is likely part of the reason. The only problem is said models require a lot of data but Humans can learn from a very small number of examples.
Humans are continuously pretrained on a variety of tasks, though. Teaching a kid to say one word takes about a year...
Re: An overview of the theory of overparameterized machine learning
#43One can say, hardwork-beats-talent in some cases and in some cases talent-beats-hardwork :P
Talent: a "smarter/clever" model Hardwork: more and more parameters
Re: An overview of the theory of overparameterized machine learning
#44A quick summary/translation for those of us who don't speak ML. We keep hearing about these giant models like GPT3 with 1.5 billion paramaters. Parameters are the things that change when we train a model, you can think about them as degrees of freedom. If you have a lot of parameters, theory made us believe that the model would just "overfit" the training data, e.g. memorize it. That's bad, because when new data come…
[deleted]
Re: An overview of the theory of overparameterized machine learning
#45I'll come out and say what's on many practicioners' minds: It could very well be that generations of academics have been WRONG about the relationship between the number of parameters (or complexity) of a model and its ability to generalize to new, previously unseen data, particularly when the data is drawn from many naturally occurring distributions -- as opposed to, say, distributions randomly chosen from the space…
[deleted]
Re: An overview of the theory of overparameterized machine learning
#46Earlier quoted context omitted.
> In practice, these huge models are, in laymans terms, fucking awesome and work really well A similarly surprising result from an adjacent community, Bayesian Statistics, is that in the case of hierarchical models, increasing your number of parameters can paradoxically reduce overfitting. The scale of parameters in Bayesian model's is no where near that of these deep neural nets, but nonetheless this is a similarly…
Do you have any good references of this phenomenon in hierarchical models?
German and Hill has a brief intro and some references.
Re: An overview of the theory of overparameterized machine learning
#47A quick summary/translation for those of us who don't speak ML. We keep hearing about these giant models like GPT3 with 1.5 billion paramaters. Parameters are the things that change when we train a model, you can think about them as degrees of freedom. If you have a lot of parameters, theory made us believe that the model would just "overfit" the training data, e.g. memorize it. That's bad, because when new data come…
> In practice, these huge models are, in laymans terms, fucking awesome and work really well A similarly surprising result from an adjacent community, Bayesian Statistics, is that in the case of hierarchical models, increasing your number of parameters can paradoxically reduce overfitting. The scale of parameters in Bayesian model's is no where near that of these deep neural nets, but nonetheless this is a similarly…
[deleted]
Re: An overview of the theory of overparameterized machine learning
#48Earlier quoted context omitted.
> In practice, these huge models are, in laymans terms, fucking awesome and work really well e.g. they generalize and work in production. No one understands why. How about the resulting weights? If most of them are close to 0, then that would mean that a part of the training is for NN to learn which of 1.5B parameters are relevant, and which are not.
There are two answers to this. First, empirically we see that the more parameters we add the better the model performs ==> Weights continue to contribute (and aren't dead) . Second, there is a very popular paper called "The lottery ticket hypothesis" [1] that in any network you can find subnetworks that work just as well. e.g. The parameters are redundant. This was written in 2018, which is a long time ago in big NN…
A couple notes...
1) Imagine the loss surface of a given model architecture; each point on the surface corresponds to a full set of weights, and the value at the point is the model loss. So, a billion-dimensional surface, give or take. There's a massive amount of flexibility in that space. Some models in the surface are sparse, but they are adjacent to models which are just as good but not sparse at all. Likewise, if you 'rotate' a sparse model, you can end up with an entirely equivalent dense model. So, you really need additional 'pressure' on the learning problem to ensure you actually get sparsity, even if the sparsity is in some sense natural.
2) IIUC, lottery ticket kinda breaks with larger models/problems. For small enough problems, the initial random projection given by the random starting weights is already good enough to build on. For bigger + more complicated problems, you need to really adapt in early training, and so lottery ticket breaks down.
Re: An overview of the theory of overparameterized machine learning
#49Earlier quoted context omitted.
My big beef with a lot of the 'leading edge' ML research is that it tends to be waaaaay too focused on classification problems, and ImageNet in particular. And, last I checked, you /do/ still fight with overfitting in classification models, by cleverly choosing learning rate schedules and using early stopping schemes, 'double descent' be damned. You can solve classification with a hash function: Hash the image, and t…
Memorization is only an issue if you allow it to be. If design the model with a "narrow" enough inner stage then that limits the level of detail (in terms of distinct representable values) passed to subsequent stages. This should give you an ML algorithm that consists of a fingerprint (approximates your hashing) stage followed by a classifier that works based on the fingerprint input (approximates a table lookup). Su…
Memorization is only an issue if you allow it to be.
Sure, it's a potential problem that can appear in the process implementing a deep learning solution. It's not an insurmountable problem. But the fact that still appears seems like an indication the situation in deep learning is more complicated than "overparameterization is not a problem".
Re: An overview of the theory of overparameterized machine learning
#50Earlier quoted context omitted.
This is likely part of the reason. The only problem is said models require a lot of data but Humans can learn from a very small number of examples.
Humans are continuously pretrained on a variety of tasks, though. Teaching a kid to say one word takes about a year...
And the very same system that is being trained to say a word is also being trained to recognize intent from intonation all the while. As soon as it can say it, the child will likely use that one word with different tones to mean different thing successfully.
We are insanely complex machines...