This is beautiful.
Wide Neural Networks of Any Architecture Are Gaussian Processes
11–20 of 36 posts
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#12Without reading the paper I bet it comes down to the central limit theorem, which itself comes down to the fact that 3rd order and higher terms don't matter asymptotically. The question is always when do you start appreciably approaching the asymptote (answer: we often have no idea). edit: re higher order terms i'm talking about the proof of the classic clt https://en.wikipedia.org/wiki/Central_limit_theorem#Proof_of…
A good paper mentioned in the sources is "Neural Tangent Kernel: Convergence and Generalization in Neural Networks". > At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods. I'm not familiar with ignoring higher order terms, besides approximations and bounds like Chebyshev's inequality and Chernoffs.
If you take the sum of a large, but finite number of random variates from a probability distribution you can show that the resulting distribution is approximately a Gaussian perturbed by Hermite polynomials. The size of these perturbations decays inversely with the order of the polynomial divided by two minus one. This is the so-called Edgeworth expansion. I found this chapter to have a good explanation of the Edgeworth expansion if you're interested in more detail: http://web.math.ku.dk/~erhansen/bootstrap_05/doku/noter/Edge...
I had a little workshop paper earlier this year showing that you can apply the Edgeworth expansion to wide, but finite neural networks: https://arxiv.org/abs/1908.10030
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#13OK, let's say NNs are GPs. What can we do with this information?
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#14OK, let's say NNs are GPs. What can we do with this information?
Probably just use a normal distribution instead of a NN.
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#15Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#16Without reading the paper I bet it comes down to the central limit theorem, which itself comes down to the fact that 3rd order and higher terms don't matter asymptotically. The question is always when do you start appreciably approaching the asymptote (answer: we often have no idea). edit: re higher order terms i'm talking about the proof of the classic clt https://en.wikipedia.org/wiki/Central_limit_theorem#Proof_of…
You're right about the central limit theorem appearing, but series expansions didn't appear; instead it is the fact that the weights are initialized to random values that seems to carry the day. I couldn't find any mention about a trained NN, this is strictly about the initial state. Yang does reference a few papers that supposedly leverage the GP correspondence to gain some insight about how to better initialize a N…
For trained wide neural networks, you can have a look at "Neural Tangent Kernel: Convergence and Generalization in Neural Networks" (https://arxiv.org/abs/1806.07572) where we explain the training of very wide ANN. This sparked a numerous amount of research and in the few last results about training wide ANN you can have a look at: https://arxiv.org/abs/1909.08156 and https://arxiv.org/abs/1904.11955
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#17OK, let's say NNs are GPs. What can we do with this information?
The paper says more than just that they are GPs, I believe it shows that the NTK (Neural Tangent Kernel https://arxiv.org/abs/1806.07572) of a large class of ANN converges at initialization. This NTK allows one to gain insight on the training of NN : speed of convergence, artifacts which can appear (checkerboard patterns can be explained with NTK), generalisation of the NN...
Would be nice to get the same result during training for all these architectures and I believe it will be the next paper of G. Yang and I eager to read it.
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#18OK, let's say NNs are GPs. What can we do with this information?
I'd guess it lets us map information processes to physical processes. Information processes follow a graph which follow a power law. Physical processes of course follow GP.
Depending on how complete the map is it may let you know us come up with 'physical' laws of information. I am rooting for something which I call Boltzmann convergence.
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#19OK, let's say NNs are GPs. What can we do with this information?
This means there is a way to convert any or perhaps just Gaussian markovian model to an ANN and vice versa.
This is interesting because markovian processes are much easier to intuit about.
Re: Wide Neural Networks of Any Architecture Are Gaussian Processes
#20OK, let's say NNs are GPs. What can we do with this information?
You can use it to estimate model uncertainty, Yarin Gal has some nice writeups on this: https://www.cs.ox.ac.uk/people/yarin.gal/website/blog_3d801a... (in this case using dropout networks as GP approximations).