41–50 of 67 posts
Earlier quoted context omitted.
What if you're not approximating a continuous function?
I think it's mostly accurate in L2 norm and not point by point. So every function in L2 can be approximated arbitrarily in L2 norm.
Earlier quoted context omitted.
What the author means by "close approximation" is clarified later (under the "Two caveats" header). The point is that you can get an arbitrarily close approximation of any continuous function, just by choosing a sufficiently large number of hidden units. That is, for any epsilon > 0, a neural network can approximate any function within epsilon of the function's exact value at all points, with enough hidden units. The…
What if you're not approximating a continuous function?
I think skilled biological actors use this overfitting to get really good at things without knowing how things actually work.
NN are still cool, though.
this is really cool
Isn't it so that all continous functions can be approximated with sums of sine waves too? That is also cool but without the AI connotations...
> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?
I wonder how efficiently it can do that compared to other systems. For example a short iterative function like this: function(complex c) complex z=0 int steps=0 while (z Can calculate with extremely high accuracy if a point in the complex plane is in the mandelbrot set or not. I would assume that a NN with the same accuracy would be of enormous size. It would probably have way more neurons then there are atoms in the…
That's one of the issues with results from theoretical math- the result is true, but it might not be useful.
As another commenter in the thread talks about, we can get a nearly identical result by using polynomials of extremely large degree, but that doesn't work well, because of overfitting. We could come up with a polynomial that also calculates with arbitrarily high accuracy if a point is in the Mandelbrot set or not, but it might end up being of 10100 degree, which is useless.
> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?
As the article says: " If you're a mathematician the argument is not difficult to follow, but it's not so easy for most people. That's a pity, since the underlying reasons for universality are simple and beautiful." Indeed, as a mathematician, the universality of neural networks is obvious to me from their definition. However, this article is explicitly not aimed at mathematicians, and (as far as I can tell) does a g…
Thinking about it a bit (I haven't finished reading the article yet)... Since the size of the hidden layer isn't specified, I suppose you could have a hidden layer node for every possible input... So, of course any function is computable with a neural network. Really the magical thing here is finding the smallest set of nodes that computes the function...
I wonder how efficiently it can do that compared to other systems. For example a short iterative function like this: function(complex c) complex z=0 int steps=0 while (z Can calculate with extremely high accuracy if a point in the complex plane is in the mandelbrot set or not. I would assume that a NN with the same accuracy would be of enormous size. It would probably have way more neurons then there are atoms in the…
I think one of the caveats was that the function should be relatively smooth and continuous, a mandelbrot set isn't very smooth is it?
You do need smoothness to prove a bound on the rate of convergence of the basis representation, and given that the boundary of the Mandelbrot doesn't have a closed form representation (as far as I'm aware), I think that the convergence of the neural network representation would be extremely slow.