Even weirdos like the Weirstrass function?
A visual proof that neural nets can compute any function
31–40 of 67 posts
Re: A visual proof that neural nets can compute any function
#32If you're interested in this you should check out Chris Eliasmith's lab - they have a Neural Engineering Framework and NENGO, which in it's current implementation is like a Python to neural network compiler.
Re: A visual proof that neural nets can compute any function
#33Re: A visual proof that neural nets can compute any function
#34Even weirdos like the Weirstrass function? https://en.wikipedia.org/wiki/Weierstrass_function
One of the caveats mentioned in the article is that the neural networks do not necessarily compute a given function precisely, but rather approximate it to an arbitrary accuracy. In the case of the Weierstrass function, this removes any weirdness. This is because, for any accuracy, you could approximate the Weierstrass function as a polynomial. Note that the polynomial example is only to show that the weirdness of the Weierstrass function is not relevant in this context. Approximating it as a polynomial is not necessarily a good approach to approximating it as a neural network.
Re: A visual proof that neural nets can compute any function
#35Earlier quoted context omitted.
What the author means by "close approximation" is clarified later (under the "Two caveats" header). The point is that you can get an arbitrarily close approximation of any continuous function, just by choosing a sufficiently large number of hidden units. That is, for any epsilon > 0, a neural network can approximate any function within epsilon of the function's exact value at all points, with enough hidden units. The…
What if you're not approximating a continuous function?
I think it's mostly accurate in L2 norm and not point by point. So every function in L2 can be approximated arbitrarily in L2 norm.
Re: A visual proof that neural nets can compute any function
#36Earlier quoted context omitted.
Ok, so by your reasoning let's have a function as a point of input, and whether it halts or not as a point of output. So now we have a function, I can't wait till "we have good techniques for constructing such a network" that maps those inputs and outputs in an abstract plane :)
That is a good example. But you are forgetting that neural networks are approximating the actual functions, so the function you described could be built with some kind of confidence level in the answer. Just like you can have some confidence that certain code will not halt from experience, neural network could also be built to do that. Not all possible functions though, unless you have infinitely large network with i…
Yeah but it's still misleading as we are talking about approximating continuous functions here, not any function. Those examples are not clearly computable, or even just continuous..
Re: A visual proof that neural nets can compute any function
#37Is anybody reading this book? How good is it for a beginner?
Re: A visual proof that neural nets can compute any function
#38Earlier quoted context omitted.
Rounding
Good example, but in this case it's actually very easy to find weights for the neural network that would do rounding with any desired precision (look for the "stack of towers" method in the article).
Works for all numbers except those ending in .5
And they are quite important to get right.
Re: A visual proof that neural nets can compute any function
#39I wonder how efficiently it can do that compared to other systems. For example a short iterative function like this: function(complex c) complex z=0 int steps=0 while (z Can calculate with extremely high accuracy if a point in the complex plane is in the mandelbrot set or not. I would assume that a NN with the same accuracy would be of enormous size. It would probably have way more neurons then there are atoms in the…
The advantage of neural networks is that they can be trained. You can give it a set of inputs and desired outputs, and do gradient descent on it.
Neural networks are essentially like trainable digital circuits. The proof of universality shows that neurons can approximate any kind of logic gate (or any input output mapping, like a lookup table.) A lookup table by itself isn't terribly useful, but you can put them in a series and make arbitrary circuits. And that makes (deep) neural networks strictly more powerful than "shallow" methods.
Re: A visual proof that neural nets can compute any function
#40I wonder how efficiently it can do that compared to other systems. For example a short iterative function like this: function(complex c) complex z=0 int steps=0 while (z Can calculate with extremely high accuracy if a point in the complex plane is in the mandelbrot set or not. I would assume that a NN with the same accuracy would be of enormous size. It would probably have way more neurons then there are atoms in the…
The advantage of neural networks is that they can be trained . You can give it a set of inputs and desired outputs, and do gradient descent on it. Neural networks are essentially like trainable digital circuits. The proof of universality shows that neurons can approximate any kind of logic gate (or any input output mapping, like a lookup table.) A lookup table by itself isn't terribly useful, but you can put them in…
A very important caveat is the ability to be trained + universality does not mean they can be trained to fit any function to arbitrary precision in finite time.