Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A visual proof that neural nets can compute any function

neuralnetworksanddeeplearning.com

Post reply on HN
Print view View original

71–80 of 86 posts

Re: A visual proof that neural nets can compute any function

#71

Post by maaku »

Tenobrus wrote: post #67
maaku wrote: post #40

Earlier quoted context omitted.

The point is more that you can't abstract neurons away into a simple "analog in, digital out" pseudo-transistor with fixed connections and expect that to describe how the brain works. The brain makes active use of all those details you are abstracting away, in ways that would make your model's predictions differ from reality.

Are you saying there is literally nothing in the brain that can be abstracted away? This seems like a very bold claim.

No, absolutely not. I'm saying that a "neural net" a la McCulloch doesn't accurately model how computation is performed by the brain. You can't accurately model human thinking by recording the brain's connections as a classical neural net. That's all I'm saying.

Re: A visual proof that neural nets can compute any function

#72

Post by ozgung »

wall_words wrote: post #61
samizdatum wrote: post #59

Earlier quoted context omitted.

The biologically-inspired nature of neural nets was a steady, mesmerizing flame for machine learning research. A chance to work on an extremely simple, provably universal system that had embarrassingly obvious rhetorical implications for profound-sounding problems like the computability of consciousness, or the computational complexity of the human brain, proved too seductive for the legions of research-moths that fl…

> but a migration of research interest away from neural nets seemed increasingly promising, and today, the migration seems largely complete. What are you talking about? Deep learning is one of the hottest areas of research today, and a lot of it has to do with neural networks. NN's are the state of the art in several domains. Case in point: http://image-net.org/challenges/LSVRC/2014/results . All of the top entries u…

Exactly. The history of NN is full of ups and downs and it's becoming increasingly popular again the form of Deep Learning thanks to increasing cloud processing power and advancements by Hinton and others. Most to of the traditional criticism of NN is related to shallow nets. But deeper and far more complex structures like those in the animal brains are not explored enough.

The next quantum leap is expected with the introduction of more specialized hardware such as neuromorphic chips: http://www.technologyreview.com/view/428235/intel-reveals-ne...

http://www.youtube.com/watch?v=pPk42xyNpSA

Re: A visual proof that neural nets can compute any function

#73

Post by sotte »

mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

If you're interested in the generalization ability of neural networks I can recommend the following paper: "Intriguing properties of neural networks" http://arxiv.org/pdf/1312.6199v4.pdf

TLDR: The authors create adversarial examples, i.e., they slightly modify the original images, which look exactly the same to humans but neural networks can't classify the images correctly anymore. What does that imply on the ability to generalize? :)

On a more general note: NN are always treated as something magic. I think a "sober view" is that NN are a special way to parameterize non-liner functions. This is neither good nor bad but it's easy to see when you look at, for example, a 2 layer NN:

$f(x) = w_2^T \sigma(W_1 \sigma (W_0 x))$

Re: A visual proof that neural nets can compute any function

#74

Post by pmelendez »

sotte wrote: post #73
mooneater wrote: post #13

Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).

If you're interested in the generalization ability of neural networks I can recommend the following paper: "Intriguing properties of neural networks" http://arxiv.org/pdf/1312.6199v4.pdf TLDR: The authors create adversarial examples, i.e., they slightly modify the original images, which look exactly the same to humans but neural networks can't classify the images correctly anymore. What does that imply on the ability…

>"which look exactly the same to humans but neural networks can't classify the images correctly anymore. What does that imply on the ability to generalize?"

Usually that means that the model is experimenting overfitting, and that's actually one challenge on any machine learning model.

Re: A visual proof that neural nets can compute any function

#75

Post by jo_ »

mturmon wrote: post #45
jo_ wrote: post #33

Earlier quoted context omitted.

You are correct. This is Cover's Theorem. "A complex pattern-classification problem, cast in a high-dimensional space nonlinearly, is more likely to be linearly separable than in a low-dimensional space, provided that the space is not densely populated." - Cover, Geometrical and Statistical properties of systems of linear inequalities with applications in pattern recognition., 1965

This is a well-known observation, but it's not at all the issue I was trying to get at, and which the existence proof of the OP raised. From the last paragraph of the paper by George Cybenko referenced in the OP: "While the approximating properties we have described are quite powerful, we have focused only on existence. The important questions that remain to be answered deal with ... how many terms in the summation (…

I misunderstood intentions. I just wanted to chime in with the name of theorem I thought you were describing. I like knowing the names of theorems, and I had a hard time finding this name when I first looked for it.

Re: A visual proof that neural nets can compute any function

#76

Post by voidlogic »

mjw wrote: post #50
voidlogic wrote: post #42

Earlier quoted context omitted.

You should explain in concrete terms why that is the case. I think its apparent that the human brain is a much more complex and advanced neural network (Intel 4004 vs Intel i7 perhaps?), but to say it is not is interesting and I would like to hear why.

Perhaps "not even close" was a bit strong, but I'm talking about a neural net as a specific mathematical model here. To say that the brain "is" an instance of a particular mathematical model isn't even really meaningful. At best you could try to argue that the brain is "modelled well by for ", although I think you'd lose that argument for all but the weakest of purposes. I'm not a neurobiologist (if you are please co…

Thank you for the explanation.

I think its worth pointing out that when computer science researchers play with different kinds of neural nets which expand beyond the simplistic version used in the OP, they still call them neural nets. I have seen abstract computer-driven neural nets that fire independently and in (more) continuous time, are plastic, lack clear layers, etc.

I think when you said: "but I'm talking about a neural net as a specific mathematical model here" that is fair, you are asserting that while this model similar to a biological one at some very primitive level the model of neural net used here is so far removed for our biological implementation as to render any comparison non-meaningful. I believe that is possible, but we still have discussed evidence that is the case.

I thought it might not be "silly" question: Consider that a turning machine simulated with pencil and paper, given enough paper and time is capable of doing any calculation your desktop computer can. Now it may take 10,000+ years and many "operators" lives, but that is besides the point.

I was thinking that the comparison between the pencil and paper turning machine and the desktop machine, might be similar to the this very basic neural net model and our own much more complex biological instance. The idea is that the two systems while varying vastly in efficiency might equivalent in what they can theoretically compute.

I don't see how asking the question of how much we can simplify our biological neural net and still have be computationally equivalent to the original (even if less efficient) is a silly question.

(I don't like silly, it implies the question is not worth asking. Calling your students question silly is a good way to make them never ask another.)

Re: A visual proof that neural nets can compute any function

#77

Post by theophrastus »

If neural nets can compute any function (as seems neatly proven here) then can they compute any function in more than a single way? If so, then upon applying the novel input (which was our goal following training) how can we know that the particular way which was computed via training set is 'right' for our novel input? If this is all true then it would seem to make neural nets perfectly unreliable as a means to modeling..?

Re: A visual proof that neural nets can compute any function

#78

Post by szabba »

jo_ wrote: post #32
akuma73 wrote: post #17

Earlier quoted context omitted.

What about XOR? From Wikipedia: In 1969 in a famous monograph entitled Perceptrons, Marvin Minsky and Seymour Papert showed that it was impossible for a single-layer perceptron network to learn an XOR function.

'Single-layer' is an easily overlooked and absolutely essential modifier. Very few networks these days are single layers. (Three is the minimum for infinite dimensional functional approximation, given nonlinear activations.) Deep Networks you see in new research papers these days have, at minimum, perhaps three layers. Some by LeCun et al. will go as high as nine or twelve layers, with four or sixteen layers in bread…

I'm pretty sure linearity was also part of the picture -- no matter how many layers of neurons you have, as long as they perform linear combinations, they could all be replaced with a single one (ignoring numeric errors).

Re: A visual proof that neural nets can compute any function

#79

Post by jo_ »

szabba wrote: post #78
jo_ wrote: post #32

Earlier quoted context omitted.

'Single-layer' is an easily overlooked and absolutely essential modifier. Very few networks these days are single layers. (Three is the minimum for infinite dimensional functional approximation, given nonlinear activations.) Deep Networks you see in new research papers these days have, at minimum, perhaps three layers. Some by LeCun et al. will go as high as nine or twelve layers, with four or sixteen layers in bread…

I'm pretty sure linearity was also part of the picture -- no matter how many layers of neurons you have, as long as they perform linear combinations, they could all be replaced with a single one (ignoring numeric errors).

I mentioned this in my post ("given nonlinear activations"), but you're correct to point out its importance. The combination of linear functions is always linear. Full stop. You need at least one non-linear layer to get reasonable results.

Re: A visual proof that neural nets can compute any function

#80

Post by mtdewcmu »

Houshalter wrote: post #69
mturmon wrote: post #60

Earlier quoted context omitted.

Yep, it's funny how different people can have different reactions. The headline of the article said "any function", and so did the first sentence of the post ("any function at all"). Later on in the article, it's qualified, but tempers are already rising. Here come people with their non-computable functions, their unmeasurable functions, their nowhere-continuous functions, all wanting to get approximated. In sup norm…

It can learn any input to output mapping. What more do you want?

The article doesn't say anything about whether it can learn it, just that it can represent it.
Post reply on HN