Earlier quoted context omitted.
I reckon piecewise continuity is enough.
Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.
[deleted]
A visual proof that neural nets can approximate any function
71–80 of 138 posts
Re: A visual proof that neural nets can approximate any function
#72People make far too big a deal of the universal function approximation property of a hidden-layer neural network. Universality should be a basic property of any decent interpolation method. Piecewise linear regression is a universal function approximator.
And GMMs and 4th order DAEs and a bunch of other things. The really interesting thing is if the sensitivity of the weights to the loss function is a simple function. Best case would be a convex optimization problem, e.g. like SVMs
Re: A visual proof that neural nets can approximate any function
#73Re: A visual proof that neural nets can approximate any function
#74Can it compute y=sin x?
Downvoted for asking a genuine question?
Re: A visual proof that neural nets can approximate any function
#75Earlier quoted context omitted.
Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.
Piecewise continuous functions can be approximated by differentiable ones. You lose some properties though. As you pointed out the derivatives might not converge, or even exist, and the limit converges pointwise at best, not uniformly.
"Piecewise continuous functions can be approximated by differentiable ones."
It depends on how you measure the distance between functions. If you are using the uniform norm (as is used in the universal approximation theorem) then this is false.
Re: A visual proof that neural nets can approximate any function
#76Great book, this is how I learned NNs
Re: A visual proof that neural nets can approximate any function
#77I think you can explain it even more clearly with smooth relative shifts, rather than sharp bump functions. I made a quick demo: https://www.desmos.com/calculator/rfaqogkbmy
Drag the sliders for w and n to change how step-like the sigmoids are and how many are combined. The purple lines are the sigmoids, relative changes at each (regularly spaced) position, which are added together to make the blue approximation to the red function. You can change the function f(x) to see how it handles other possibilities, including piecewise/discontinuous ones like "{x=5: x}".
Re: A visual proof that neural nets can approximate any function
#78Can it compute y=sin x?
Downvoted for asking a genuine question?
I think it's because it looks like you didn't read the article, which explains repeatedly that this is a theorem of universality. Any continuous function will work, and sin is continuous.
Check this out: https://www.desmos.com/calculator/rfaqogkbmy
Try changing f(x) to "sin(x)" or whatever you like (e.g. sin(10x) may be more interesting). Then drag the sliders for w (how step-like the purple pieces are) and n (how many pieces to add together). The function f(x) is red, and the approximation is blue.
Re: A visual proof that neural nets can approximate any function
#79Skimming through the article, I understand that the author... A) was focusing on functions that take a certain amount of input variables and B) that the function (that s/he mirrored using the neural net) computes out of it directly one or more of result(s). C) To do that s/he used a backpropagation network (which is the only model I know very well). Right or wrong? EDIT: when I say "directly" I mean that the function…
If you want to know the details, you should read the article more thoroughly. There is no back-propagation (or any training) involved, as this article is about what kinds of things neural networks can do in principle. I.e. how we can be sure that neural networks can in theory solve some problem we have. In practice, you have to actually find a network (by training) that solves your problem with a reasonable amount of resources, using the data you have, and that's a whole other issue (in fact, an entire research field!).
Re: A visual proof that neural nets can approximate any function
#80Earlier quoted context omitted.
I reckon piecewise continuity is enough.
Neural networks can't be discontinuous, even piecewise, because you wouldn't be able to differentiate at the discontinuities.
Do neural nets need to be continuous functions of the inputs, or of the parameters? My recollection is that it only has to be the latter.