Any Deep ReLU Network Is Shallow
arxiv.org
Any Deep ReLU Network Is Shallow
1–10 of 62 posts
Re: Any Deep ReLU Network Is Shallow
#2Nice! I have to admit I didn't follow all the math, but the layers get really wide, right? You still need to describe all the partitions between the pieces of the piecewise linear functions. Does this save any computation at inference time?
Re: Any Deep ReLU Network Is Shallow
#3I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢf(aᵢx+bᵢ)+dᵢ. You just need an infinite number of them. This sum is a neural network with a single hidden layer.
Re: Any Deep ReLU Network Is Shallow
#4So any N-layer network with the relu _/ activation function can be restated in terms of a 3 layer network as long as the 3 layers can have infinite parameters and as long as you have near infinite computing power to convert the big ones. Cool but not yet useful.
Re: Any Deep ReLU Network Is Shallow
#5I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…
Thanks for the explanation. I've heard this before but think I get it now!
Re: Any Deep ReLU Network Is Shallow
#6I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…
> Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network.
I’m out of date on the research, but I suspect the real value here is the algorithm. Sounds like some version of this could eventually help reduce inference time
Re: Any Deep ReLU Network Is Shallow
#7So any N-layer network with the relu _/ activation function can be restated in terms of a 3 layer network as long as the 3 layers can have infinite parameters and as long as you have near infinite computing power to convert the big ones. Cool but not yet useful.
> How well does a classic deep net architecture like AlexNet or VGG19 classify on a
standard dataset such as CIFAR-10 when its “width”— namely, number of channels
in convolutional layers, and number of nodes in fully-connected internal layers —
is allowed to increase to infinity? Such questions have come to the forefront in the
quest to theoretically understand deep learning and its mysteries about optimization
and generalization. They also connect deep learning to notions such as Gaussian
processes and kernels. A recent paper [Jacot et al., 2018] introduced the Neural
Tangent Kernel (NTK) which captures the behavior of fully-connected deep nets in
the infinite width limit trained by gradient descent; this object was implicit in some
other recent papers. An attraction of such ideas is that a pure kernel-based method
is used to capture the power of a fully-trained deep net of infinite width.
https://openreview.net/pdf?id=rkl4aESeUH, https://github.com/google/neural-tangents
> It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width.
https://arxiv.org/abs/1711.00165
And of course, one needs to look back at SVMs applying a kernel function and separating with a line, which looks a lot like an ANN with a single hidden layer followed by a linear mapping.
https://stats.stackexchange.com/questions/238635/kernel-meth...
Re: Any Deep ReLU Network Is Shallow
#8The benefits here are not really to do with implementing inference, but rather the improvement in explainability that a shallow network can provide. Skip to section 5 (page 8) for that.
Re: Any Deep ReLU Network Is Shallow
#9I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…
> Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. I’m out of date on the research, but I suspect the real value here is the algorithm. Sounds like some version of this could eventually help reduce inference time
I suspect the trade-off for a shallow network is an exponential explosion in weights and computation.
Re: Any Deep ReLU Network Is Shallow
#10I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…
This is like how you can write any boolean function as a (large) sum of minterms.