Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Any Deep ReLU Network Is Shallow

arxiv.org

Post reply on HN
Print view View original

21–30 of 62 posts

Re: Any Deep ReLU Network Is Shallow

#21

Post by throwawaymaths »

WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

this is the "universal approximator theorem".

https://cognitivemedium.com/magic_paper/assets/Hornik.pdf

> infinite number of [activations]

I don't think you need an infinite number of them, there is a relationship between "how close you want to get" and "how many of them you need".

Re: Any Deep ReLU Network Is Shallow

#22

Post by throwawaymaths »

WithinReason wrote: post #9
curiousllama wrote: post #6

Earlier quoted context omitted.

> Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. I’m out of date on the research, but I suspect the real value here is the algorithm. Sounds like some version of this could eventually help reduce inference time

I suspect the trade-off for a shallow network is an exponential explosion in weights and computation.

I'm not sure it's exponential, but it is dramatic.

Re: Any Deep ReLU Network Is Shallow

#23

Post by throwawaymaths »

kixiQu wrote: post #8

The benefits here are not really to do with implementing inference, but rather the improvement in explainability that a shallow network can provide. Skip to section 5 (page 8) for that.

If anything inference will probably get worse. Typically in ML constraining your architecture (which is what you're doing when you make a deep narrow network out of a shallow wide network, or rnns or grus/lstms, or transformers out of fcnns) yields inference and training performance benefits.

Re: Any Deep ReLU Network Is Shallow

#24

Post by visarga »

superkuh wrote: post #4

So any N-layer network with the relu _/ activation function can be restated in terms of a 3 layer network as long as the 3 layers can have infinite parameters and as long as you have near infinite computing power to convert the big ones. Cool but not yet useful.

> So any N-layer network with the relu _/ activation function

The paper talks only about models with additive operations among activations. It doesn't say anything about more complex networks like transformers.

In transformers there are multiplicative interactions between activations inside the attention matrix, it is unclear if they can be approximated with just a 3 layer ReLU network, or if such conversion would be practical at all.

Re: Any Deep ReLU Network Is Shallow

#25

Post by wewxjfq »

WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

This is such a bullshit HN comment, lmao.

Re: Any Deep ReLU Network Is Shallow

#26

Post by dang »

wewxjfq wrote: post #25
WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

This is such a bullshit HN comment, lmao.

Could you please stop posting unsubstantive comments and flamebait? You've unfortunately been doing it repeatedly. It's not what this site is for, and destroys what it is for.

If you know more than others, that's great—please share some of what you know, so the rest of us can learn something, or else don't post. Sneers and putdowns only make everything worse.

If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and taking the intended spirit of the site more to heart, we'd be grateful.

Re: Any Deep ReLU Network Is Shallow

#27

Post by joe_the_user »

WithinReason wrote: post #9
curiousllama wrote: post #6

Earlier quoted context omitted.

> Based on this proof, we provide an algorithm that, given a deep ReLU network, finds the explicit weights of the corresponding shallow network. I’m out of date on the research, but I suspect the real value here is the algorithm. Sounds like some version of this could eventually help reduce inference time

I suspect the trade-off for a shallow network is an exponential explosion in weights and computation.

From the paper:

    Corollary 1. Given a ReLU network N : Rn → Rm, the shallow 
    network S : Rn → Rm of depth L = 3, as specified in the proof 
    of Theorem 1, has width bounded by 
    max{2n + k, 2n + p, 2 · p · m}
I think p might be the number of layers but I'm not sure and I'm sure about k at all.

So it they might be claiming a pretty tight bound in on weights.

Re: Any Deep ReLU Network Is Shallow

#28

Post by blt »

WithinReason wrote: post #3

I thought it was known for a long time that any function can be represented with a neural network with a single layer. It's an almost trivial finding if you think about it: imagine you have a steep step function that looks something like this: __/^^ with the non-zero derivative in a small range, e.g. 0.000-0.001 (or ϵ if you like). Let's call this f. You can piece together any function from these tiny pieces as ∑ᵢ cᵢ…

The result in the paper is not an approximation result. The shallow network is of bounded width and is exactly equal to the deep one.
Post reply on HN