Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can compute any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

21–30 of 67 posts

Re: A visual proof that neural nets can compute any function

#21

Post by throwaway39202 »

JumpCrisscross wrote: post #18

> No matter what the function, there is guaranteed to be a neural network so that for every possible input, x, the value f(x) (or some close approximation) is output from the network Love NNs, but accuracy is relative. At some resolution of "close approximation," every function "computes" every other function.

What the author means by "close approximation" is clarified later (under the "Two caveats" header). The point is that you can get an arbitrarily close approximation of any continuous function, just by choosing a sufficiently large number of hidden units. That is, for any epsilon > 0, a neural network can approximate any function within epsilon of the function's exact value at all points, with enough hidden units. The smaller the epsilon, the better the approximation you want, and the more hidden units you need.

Re: A visual proof that neural nets can compute any function

#22

Post by p1esk »

gizmo686 wrote: post #5
p1esk wrote: post #4

Earlier quoted context omitted.

What would be an example of a practical task that needs to be modeled with such a function?

Rounding

Good example, but in this case it's actually very easy to find weights for the neural network that would do rounding with any desired precision (look for the "stack of towers" method in the article).

Re: A visual proof that neural nets can compute any function

#24

Post by gizmo686 »

p1esk wrote: post #22
gizmo686 wrote: post #5

Earlier quoted context omitted.

Rounding

Good example, but in this case it's actually very easy to find weights for the neural network that would do rounding with any desired precision (look for the "stack of towers" method in the article).

Yeah, I think I picked rounding because the article primed me to think about it (the proof presented is essentially based on the ability to create steps, and rounding is my go-to example of a step function). However, any attempt to construct such a network would have a finite amount of steps in it, while the actual step function would have infinite.

In practice, if we needed such a network, we would probably have a restricted domain, so there is only a finite number of discontinuous, and there would be a sufficiently large network that could solve it to an arbitrary precision.

We would not be able to do this with more exotic functions that are densly discontinues, such as the function:

    f(x) = 0  iff x is rational
    f(x) = 1  iff x is irrational
There is no way a neural network can reasonably model that function.

Re: A visual proof that neural nets can compute any function

#26

Post by bluesign »

Isn't this similar to infinite monkey theorem: "Given an infinite length of time, a chimpanzee punching at random on a typewriter would almost surely type out all of Shakespeare's plays."

Given that we can have infinite big neural network and train it in an infinite length of time, it can compute any function.

Re: A visual proof that neural nets can compute any function

#27

Post by otabdeveloper1 »

> Every continuous function in the function space can be represented as a linear combination of basis functions

https://en.wikipedia.org/wiki/Basis_function

This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?

Re: A visual proof that neural nets can compute any function

#28

Post by gizmo686 »

otabdeveloper1 wrote: post #27

> Every continuous function in the function space can be represented as a linear combination of basis functions https://en.wikipedia.org/wiki/Basis_function This is basic and obvious math. Does slapping the word 'neural' magically make obvious results 1000% more interesting? Why? Because the word 'neural' carries some of that artificial-intelligence-technology-of-the-future cachet?

As the article says:

" If you're a mathematician the argument is not difficult to follow, but it's not so easy for most people. That's a pity, since the underlying reasons for universality are simple and beautiful."

Indeed, as a mathematician, the universality of neural networks is obvious to me from their definition. However, this article is explicitly not aimed at mathematicians, and (as far as I can tell) does a good job of presenting the argument without requiring unnecessary math knowledge, which is something that we tend to be bad at doing.

Furthermore, the universality does not directly follow from the quote you provide, as you would still need to show that the hidden layer neurons form a basis. Infact, the article was about constructing such a basis in the neurons. The only thing that bringing in your quote would serve to do is make the article more confusing by unnecessarily introducing the concept of basis functions.

Re: A visual proof that neural nets can compute any function

#29

Post by TheOtherHobbes »

throwaway39202 wrote: post #21
JumpCrisscross wrote: post #18

> No matter what the function, there is guaranteed to be a neural network so that for every possible input, x, the value f(x) (or some close approximation) is output from the network Love NNs, but accuracy is relative. At some resolution of "close approximation," every function "computes" every other function.

What the author means by "close approximation" is clarified later (under the "Two caveats" header). The point is that you can get an arbitrarily close approximation of any continuous function, just by choosing a sufficiently large number of hidden units. That is, for any epsilon > 0, a neural network can approximate any function within epsilon of the function's exact value at all points, with enough hidden units. The…

What if you're not approximating a continuous function?

Re: A visual proof that neural nets can compute any function

#30

Post by tfgg »

bluesign wrote: post #26

Isn't this similar to infinite monkey theorem: "Given an infinite length of time, a chimpanzee punching at random on a typewriter would almost surely type out all of Shakespeare's plays." Given that we can have infinite big neural network and train it in an infinite length of time, it can compute any function.

I don't think so, since the former is about the occurrence of sequences in stochastic, ergodic processes, and the latter is about approximating deterministic functions.

It's more like given an infinite number of parameters, you have enough degrees of freedom to describe the vector space a function lives in with some set of basis functions, and a neural net supplies that.

Post reply on HN