Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can approximate any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

61–70 of 138 posts

Re: A visual proof that neural nets can approximate any function

#61

Post by throwawaymath »

braised_babbage wrote: post #30
throwawaymath wrote: post #22

Earlier quoted context omitted.

While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. A neural network computes linear transformations, so unless I'm missing something I'm a little surp…

One of the things about continuous functions is that they aren't necessarily differentiable, cf. https://en.wikipedia.org/wiki/Weierstrass_function

True, but that was kind of an afterthought of my point. All continuous functions still comprise a vector space.

Re: A visual proof that neural nets can approximate any function

#62

Post by throwawaymath »

tnecniv wrote: post #33
throwawaymath wrote: post #22

Earlier quoted context omitted.

While we're at it, doesn't the "universality theorem" (as the article calls it) basically follow immediately from the fact that the set of all continuous functions comprises a vector space? If the continuous function is additive, it's linear. If it's nonlinear, you can differentiate it to obtain a linear approximation. A neural network computes linear transformations, so unless I'm missing something I'm a little surp…

Continuous != differentiable. The standard proof uses some functional analysis techniques but nothing too complicated to show you can get arbitrarily close to any continuous function with an NN. That includes things like step functions whose derivatives are not defined everywhere.

You could also just cite the Weierstrass function, which is continuous everywhere and differentiable nowhere. But that's separate from the meat of my point, which is that C(R) is a vector space.

Re: A visual proof that neural nets can approximate any function

#65

Post by edflsafoiewq »

whatshisface wrote: post #44
contravariant wrote: post #42

Earlier quoted context omitted.

I reckon piecewise continuity is enough.

Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.

If that was the reason, the argument would demand the function be differentiable, not just continuous, wouldn't it?

Re: A visual proof that neural nets can approximate any function

#67

Post by sgillen »

a-nikolaev wrote: post #31

Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).

Feedforward neural networks are not turing complete but recurrent neural networks are.

Re: A visual proof that neural nets can approximate any function

#68

Post by [deleted] »

sgillen wrote: post #67
a-nikolaev wrote: post #31

Approximate, not compute. The function also must be continuous. NNs are good for approximation / interpolation / extrapolation, which makes them quite useful for certain domains of problems. But of course, it does not make them a kind of universal computing machine (in the computability sense, like universal Turing machines).

Feedforward neural networks are not turing complete but recurrent neural networks are.

[deleted]

Re: A visual proof that neural nets can approximate any function

#69

Post by contravariant »

whatshisface wrote: post #44
contravariant wrote: post #42

Earlier quoted context omitted.

I reckon piecewise continuity is enough.

Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.

Piecewise continuous functions can be approximated by differentiable ones. You lose some properties though. As you pointed out the derivatives might not converge, or even exist, and the limit converges pointwise at best, not uniformly.

Re: A visual proof that neural nets can approximate any function

#70

Post by soVeryTired »

People make far too big a deal of the universal function approximation property of a hidden-layer neural network. Universality should be a basic property of any decent interpolation method.

Piecewise linear regression is a universal function approximator.

Post reply on HN