I don't quite follow what is a Gaussian process. What do you use, node weights, node inputs/outputs, or...?
Wide Neural Networks of Any Architecture Are Gaussian Processes
21–30 of 36 posts
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#22Earlier quoted context omitted.
You're right about the central limit theorem appearing, but series expansions didn't appear; instead it is the fact that the weights are initialized to random values that seems to carry the day. I couldn't find any mention about a trained NN, this is strictly about the initial state. Yang does reference a few papers that supposedly leverage the GP correspondence to gain some insight about how to better initialize a N…
Yes. I will have things to say about training, but that requires building up some theoretical foundations. This paper is the first step in laying it out. Stay tuned! :)
So do you already have any intuition of what training does to the initial GP? Obviously the training adjust the various weights in complicated ways, which to me feels like it should correspond to some sort of marginalisation on the GP, but I'm not really aware if that's a thing people do (undoubtedly someone has tried it though).
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#23So, if I understand this correctly...
Gaussian Processes are a way of trying every possible function to fit a set of data points, being constrained a bit more with each new data point.
All neural networks in the list, given sufficient size, are essentially Gaussian in their behavior, and thus share the same features and limitations.
Right?
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#24Good results often seem "obvious" in ex post. So take this as a compliment.
Intuitively (I have never read a paper in this field), since you are talking about wide networks, I also expected that a CLT would be used. For "dense" layers it is pretty obvious that one should be able to characterize each layer aggregation based on a CLT, and so forth. Some sort of mild independence assumption, mixing or martingales, on the sampling should be sufficient. I think therefore the goal of a paper for any architecture would be to figure out a way to generalize this for different layers.
However, one thing I notice is that you seem to assume that weights (or whatever is initialized) are initialized with a Gaussian distribution?
That seems a bit restricted. The appeal of this approach with wide networks, I think, is that any independent initialization of weights would lead to transformations of GP.
Perhaps I am misunderstanding also the implications. Could you generally trace a dependence onto the distribution of the last layer weights, even if they are not normal? Or do you need the GP for your conditioning?
On the one hand, Gaussian initialization is I think not really encompassing "all architectures" as practically used, but more specifically, it seems that the end goal of this research program would be to generalize beyond this (much like in regression, one uses CLTs exactly to get away from parametric assumptions). Or is that where you plan to go?
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#25Good results often seem "obvious" in ex post. So take this as a compliment. Intuitively (I have never read a paper in this field), since you are talking about wide networks, I also expected that a CLT would be used. For "dense" layers it is pretty obvious that one should be able to characterize each layer aggregation based on a CLT, and so forth. Some sort of mild independence assumption, mixing or martingales, on th…
Reading a bit more, it's really interesting that your result relies on Lemma G.4, which is a CLT based on independence or mixing (as you wish), whereas all theorems assume that anything put in is Gaussian.
It "smells" like that is, or should not be necessary.
The elegance of this approach, and GP in general, is that you use scale and independence and get to a specific distribution. Therefore, assuming that inputs are Gaussian seems restrictive. In some appropriate sense, it should not be required.
But again, I am probably misreading something.
What I would be looking for as a referee is: "Based on ANY random initialization (mild independence condition), it holds that wide networks become Gaussian"
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#26This sounds important and interesting but isn't wide the key word here? They talk about shallow NNs and deep fully connected NNs but that would seem to leave out a lot. I mean, the article puts forward a distinct language/model to expression neural nets in, which is cool but are they talking about all or most of the NNs you see today? If so, huge but still. Fo-get-a-bout-it, see SiempreViernes' comment: "I couldn't f…
Hi, the author here. Thanks for your interest! Let me try answering some of your questions. > This sounds important and interesting but isn't wide the key word here? Yes, width is very important for this result. Given the size of modern deep neural networks, I (and most people in the deep learning theory community, by now) believe the large width regime is the appropriate regime to study neural networks. > I mean, th…
> Try throw me an architecture and watch if I can't throw you back a GP :)
On the pragmatic side, would that GP train faster than the NN? In my little experimentation with GPs, I found them awfully slow. However, maybe what I tried (it was black box for me) used some brute force approach, and there are other more fine-tuned algorithms. Since you are an expert in the area, what's your take?
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#27OK, let's say NNs are GPs. What can we do with this information?
You can use it to estimate model uncertainty, Yarin Gal has some nice writeups on this: https://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801a... (in this case using dropout networks as GP approximations).
How would we use a property of networks with random weights to estimate uncertainty of trained models in which which the weights are (as much as we can) trained to be not random?
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#28Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#29Earlier quoted context omitted.
Yes. I will have things to say about training, but that requires building up some theoretical foundations. This paper is the first step in laying it out. Stay tuned! :)
So do you already have any intuition of what training does to the initial GP? Obviously the training adjust the various weights in complicated ways, which to me feels like it should correspond to some sort of marginalisation on the GP, but I'm not really aware if that's a thing people do (undoubtedly someone has tried it though).
Well, if there's a 1-1 map between GP and NNs, shouldn't we be able to determine the effect of a gradient descent step on a GP by combining the two maps?
I have only barely glanced at the paper mind-you, so I couldn't say the details but still.
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#30Earlier quoted context omitted.
Yes. I will have things to say about training, but that requires building up some theoretical foundations. This paper is the first step in laying it out. Stay tuned! :)
So do you already have any intuition of what training does to the initial GP? Obviously the training adjust the various weights in complicated ways, which to me feels like it should correspond to some sort of marginalisation on the GP, but I'm not really aware if that's a thing people do (undoubtedly someone has tried it though).
@fgabriel mentioned this below: if the network is parametrized in a certain way, then the GP evolves according to a linear equation (if trained with square loss). In this linear equation, a different kernel shows up, known as the Neural Tangent Kernel. An intuitive way to think about this is to Taylor expand the parameters-to-function map around the initial set of parameters: f = f_0 + J d\theta, where J is the Jacobian of the neural network function against the parameters. Following this logic, the change in parameters affects the neural network function roughly linearly, as long as the parameters don't venture too far away from their original values. The Neural Tangent Kernel is then given by JJ^T.
In addition to the paper mentioned by @fgabriel, this paper [1] explains it in more detail as well, and the equations you are looking for are 14, 15, and 16.