Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can compute any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

11–20 of 86 posts

Re: A visual proof that neural nets can compute any function

#11

Post by gphil »

voidlogic wrote: post #7
tomp wrote: post #4

As mentioned in the article, the formal statement is actually "neural nets can approximate (arbitrarily well, using the supremum metric) any continuous function". For other norms, it can also approximate non-continuous functions.

It would have been pretty interesting if this had NOT held. It would have meant that even though "neural nets can NOT approximate (arbitrarily well, using the supremum metric) any continuous function", a neural network (the humans involved) was able to discover this limitation. I find the idea of a neural net finding a limitation of a neural net, to be interesting.

More generally, logic can be used to demonstrate the limitations of logic:

https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th...

You could say that a neural network found this limitation of neural networks, to the extent that neural networks could be defined in terms mathematical logic. However, it's not guaranteed that the neural networks in our brain could be explained in these terms--the physical processes underlying them are not fully understood.

Re: A visual proof that neural nets can compute any function

#12

Post by jbarrow »

jokoon wrote: post #8

who's to doubt neural networks are completely awesome ? Any more news about those chips that were optimized for neural networks ? Was it IBM or Samsung ?

It was the IBM SyNAPSE Chip [1]. It's actually been known since 1988/89 that neural networks can approximate any continuous function [2], but this chapter explains how in a much more intuitive sense.

[1] http://www.research.ibm.com/cognitive-computing/neurosynapti... [2] http://dl.acm.org/citation.cfm?id=70408

Re: A visual proof that neural nets can compute any function

#14

Post by mjw »

voidlogic wrote: post #7
tomp wrote: post #4

As mentioned in the article, the formal statement is actually "neural nets can approximate (arbitrarily well, using the supremum metric) any continuous function". For other norms, it can also approximate non-continuous functions.

It would have been pretty interesting if this had NOT held. It would have meant that even though "neural nets can NOT approximate (arbitrarily well, using the supremum metric) any continuous function", a neural network (the humans involved) was able to discover this limitation. I find the idea of a neural net finding a limitation of a neural net, to be interesting.

Humans are not neural networks in the formal sense used here, not even close.

Re: A visual proof that neural nets can compute any function

#15

Post by mturmon »

tomp wrote: post #4

As mentioned in the article, the formal statement is actually "neural nets can approximate (arbitrarily well, using the supremum metric) any continuous function". For other norms, it can also approximate non-continuous functions.

You also need the qualifier "...any continuous function, on a compact set."

Once you add all three qualifiers in (approximate/continuous/compact) it starts to sound more like math and less like a miracle.

Incidentally, one thing of great interest is, how does the number of hidden units required behave as a function of dimensionality of the input domain. In dramatic language, "Can neural networks get around the curse of dimensionality?"

The Cybenko proof does not give enlightenment about that question. Andrew Barron (http://www.stat.yale.edu/~arb4/) had some results that seemed to indicate the dependence was moderate (not exponential). I'm not aware what the state of the art currently is.

Re: A visual proof that neural nets can compute any function

#17

Post by akuma73 »

mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

What about XOR?

From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

Re: A visual proof that neural nets can compute any function

#18

Post by JoeAltmaier »

akuma73 wrote: post #17
mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

What about XOR? From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

Neural nets now use non-linear functions so they can do things like xor.

Re: A visual proof that neural nets can compute any function

#19

Post by mjw »

mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

Quite. It's not hard to come up with models or families of functions which share this property.

What matters is not only whether they can learn it but how much data they need to learn it to a given degree of accuracy. This is the kind of question addressed by nonparametric statistics and statistical learning theory.

Re: A visual proof that neural nets can compute any function

#20

Post by pesenti »

akuma73 wrote: post #17
mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

What about XOR? From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

http://en.wikipedia.org/wiki/Feedforward_neural_network#medi...
Post reply on HN