Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Any Deep ReLU Network Is Shallow

arxiv.org

Post reply on HN
Print view View original

31–40 of 62 posts

Re: Any Deep ReLU Network Is Shallow

#31

Post by hervature »

WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

I would just add the caveat that it is a single hidden layer. If you were to write this as a sequential model you would have something like:

nn.Sequential(

    Dense(many neurons),

    Dense(1)
)

From this, it's pretty easy to see it is "two" layers but also from your equation c_i and a_i denote two separate matrix multiplications.

Re: Any Deep ReLU Network Is Shallow

#32

Post by littlestymaar »

foota wrote: post #18

I'm not an expert, but I wonder if the wide network could then be trimmed down for special purposes? E.g., find the parts of the network needed for a certain task by looking at activations on inputs and dump the rest.

That was my thought as well, it would be nice if someone more knowledgeable came and explained us why it's stupid/doesn't work

Re: Any Deep ReLU Network Is Shallow

#33

Post by neodypsis »

foota wrote: post #18

I'm not an expert, but I wonder if the wide network could then be trimmed down for special purposes? E.g., find the parts of the network needed for a certain task by looking at activations on inputs and dump the rest.

Weight pruning is already a thing. See, for example: https://www.tensorflow.org/model_optimization/guide/pruning

Re: Any Deep ReLU Network Is Shallow

#35

Post by coderenegade »

curiousllama wrote: post #6
WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

> Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. I’m out of date on the research, but I suspect the real value here is the algorithm. Sounds like some version of this could eventually help reduce inference time

If you can go back and forth between representations with better than quadratic scaling, it would mean RNNs with infinite context lengths and no vanishing gradient. You wouldn't need transformers.

Re: Any Deep ReLU Network Is Shallow

#36

Post by elcomet »

blt wrote: post #28
WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

The result in the paper is not an approximation result. The shallow network is of bounded width and is exactly equal to the deep one.

Only for relu networks though, which are already piecewise linear functions.

Re: Any Deep ReLU Network Is Shallow

#37

Post by mkl »

charliea0 wrote: post #15
madmaxmcfly wrote: post #12

Earlier quoted context omitted.

Isn't that just like a Taylor series?

Yes - or Fourier series or other decomposition into a sum of orthogonal basis functions.

I don't think orthogonality is important here. Taylor series basis functions are not orthogonal, and nor are ReLUs.

Re: Any Deep ReLU Network Is Shallow

#40

Post by hedgehog »

foota wrote: post #18

I'm not an expert, but I wonder if the wide network could then be trimmed down for special purposes? E.g., find the parts of the network needed for a certain task by looking at activations on inputs and dump the rest.

Besides pruning (which is what you describe) you might find the "lottery ticket hypothesis" interesting. The idea is essentially that in a randomly initialized large network there is a subset of weights that are already closeish for a given task and training helps tune that and suppress everything else, and that knowing this you can actually get better results by pruning before training.
Post reply on HN