41–50 of 86 posts
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
Earlier quoted context omitted.
It would have been pretty interesting if this had NOT held. It would have meant that even though "neural nets can NOT approximate (arbitrarily well, using the supremum metric) any continuous function", a neural network (the humans involved) was able to discover this limitation. I find the idea of a neural net finding a limitation of a neural net, to be interesting.
Humans are not neural networks in the formal sense used here, not even close.
Earlier quoted context omitted.
If you join a bunch of perceptrons together that limitation goes away. Another path is to make the problem effectively linear again by transforming into higher dimensions, kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly.
> kernels do this with one clever trick that allows them to avoid the computational cost of doing so explicitly Minsky and Papert hate him! Use this one weird trick …. (Sorry; I couldn't resist!)
Earlier quoted context omitted.
It would have been pretty interesting if this had NOT held. It would have meant that even though "neural nets can NOT approximate (arbitrarily well, using the supremum metric) any continuous function", a neural network (the humans involved) was able to discover this limitation. I find the idea of a neural net finding a limitation of a neural net, to be interesting.
More generally, logic can be used to demonstrate the limitations of logic: https://en.wikipedia.org/wiki/G%C3%B6del's_incompleteness_th... You could say that a neural network found this limitation of neural networks, to the extent that neural networks could be defined in terms mathematical logic. However, it's not guaranteed that the neural networks in our brain could be explained in these terms--the physical process…
This is of course true, but nevertheless it is an interesting result (hence why we study Gödel's incompleteness theorem).
Earlier quoted context omitted.
> how does the number of hidden units required behave as a function of dimensionality of the input domain If I recall correctly, a non-linear problem can be solved as a linear problem if you consider more dimensions. The hidden layer add dimensions. So, it's not a function of the input domain but of the problem domain, which usually isn't explicitly known.
You are correct. This is Cover's Theorem. "A complex pattern-classification problem, cast in a high-dimensional space nonlinearly, is more likely to be linearly separable than in a low-dimensional space, provided that the space is not densely populated." - Cover, Geometrical and Statistical properties of systems of linear inequalities with applications in pattern recognition., 1965
From the last paragraph of the paper by George Cybenko referenced in the OP:
"While the approximating properties we have described are quite powerful, we have focused only on existence. The important questions that remain to be answered deal with ... how many terms in the summation (or equivalently, how many neural nodes) are required to yield an approximation of a given quality? [...] We suspect quite strongly that the overwhelming majority of approximation problems will require astronomical numbers of terms. This feeling is based on the curse of dimensionality that plagues multidimensional approximation theory and statistics."
[I'm highly familiar with both the paper by Cybenko, and the paper by Tom Cover on linear separability that is the source of the Wikipedia quote pasted above, having gone through them carefully as part of my PhD. Andrew Barron, mentioned above, was Tom Cover's student, and the work Andrew did can be viewed as another approach to this problem.]
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
It computes because it is detached, there is no feeling. To learn, we would need to first create something that is conscious, an ever-changing self. It needs to be in reference to itself, not some notion of compute in a detached paradigm.
> "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E". This definition is notable for its defining machine learning in fundamentally operational rather than cognitive terms, thus following Alan Turing's proposal in Turing's paper "Computing Machinery and Intelligence" that the question "Can machines think?" be replaced with the question "Can machines do what we (as thinking entities) can do?"
It's important, because we don't need to care about cognition or consciousness, and we can still write programs that solve problems well by making inferences from patterns in data.
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
Seems to me that jumping to whether they can "learn any function" as you asked, would certainly have come after this is first established.
http://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theor...
Earlier quoted context omitted.
Humans are not neural networks in the formal sense used here, not even close.
You should explain in concrete terms why that is the case. I think its apparent that the human brain is a much more complex and advanced neural network (Intel 4004 vs Intel i7 perhaps?), but to say it is not is interesting and I would like to hear why.
To put it simply, it is possible for ANNs to fail at something and for us to succeed at the same thing.
Earlier quoted context omitted.
Humans are not neural networks in the formal sense used here, not even close.
You should explain in concrete terms why that is the case. I think its apparent that the human brain is a much more complex and advanced neural network (Intel 4004 vs Intel i7 perhaps?), but to say it is not is interesting and I would like to hear why.
I'm not a neurobiologist (if you are please correct/clarify!), but here are just a few of doubtless many significant differences as I understand them:
* Real neurons don't update in a bunch of coordinated discrete timesteps, as an ANN learning algorithm does. They can fire independently and in continuous time.
* Real neurons' activation behaviour is closer to sudden delta-function-like spikes, rather than a smooth activation function or something which allows them to stay in a firing state for longer than a short pulse.
* The structure of the connection graph of neurons in the brain is incredibly more complex than that of artificial neural net models, can change over time, isn't split into obvious layers from input to output, there isn't a clearly-identifiable error signal which is used to train it at the outputs, ...
* A real neuron is an incredibly complex biological system whose behaviour can be modulated in all sorts of ways and which I expect would take a nightmarishly complicated set of PDEs to model with any degree of realism even as an isolated unit. To think we've captured all aspects of their behaviour relevant to human cognition with a simple weighted sum and a sigmoid (say) seems pretty naive.
Details aside though, the idea that you can start drawing deep philosophical conclusions about the nature of human thought, our ability to conceive of our own limitations etc, based on an analogy between a complex biological system and a simple mathematical formalism which is at best loosely influenced by certain limited aspects of it -- it's just silly, one of those "not even wrong" sort of statements.