Any Deep ReLU Network Is Shallow
arxiv.org
Any Deep ReLU Network Is Shallow
1–6 of 6 posts
Re: Any Deep ReLU Network Is Shallow
#2It doesn’t surprise me. It’s been known a long time that you can model arbitrary functions with a 3-layer network with logistic activation.
Re: Any Deep ReLU Network Is Shallow
#3It doesn’t surprise me. It’s been known a long time that you can model arbitrary functions with a 3-layer network with logistic activation.
Do you have any further information or a source for this? As someone unfamiliar with ML, this sounds crazy to me.
Re: Any Deep ReLU Network Is Shallow
#4It seems intuitive since ReLU is just a type of implicit regularization. Why would subsequent gradient descent help once you've achieved the benefit of throwing away the "outliers" or data beyond the threshold you want?
Re: Any Deep ReLU Network Is Shallow
#5It doesn’t surprise me. It’s been known a long time that you can model arbitrary functions with a 3-layer network with logistic activation.
Do you have any further information or a source for this? As someone unfamiliar with ML, this sounds crazy to me.
It was first proposed here[1]: "Approximation by superpositions of a sigmoidal function"
[1]: https://link.springer.com/content/pdf/10.1007/BF02551274.pdf
Re: Any Deep ReLU Network Is Shallow
#6don't confuse this with universal approximation - yes shallow ReLU networks are dense in functional space, so at the limit you should be able to get any function you want - but they are talking about exact representation with finitely many neurons here.