51–60 of 86 posts
Can't the same be said for Fourier series, which make no claims to be some kind of AI? And likewise humble polynomials: http://en.wikipedia.org/wiki/Stone%E2%80%93Weierstrass_theor...
As mentioned in the article, the formal statement is actually "neural nets can approximate (arbitrarily well, using the supremum metric) any continuous function". For other norms, it can also approximate non-continuous functions.
You also need the qualifier "...any continuous function, on a compact set. " Once you add all three qualifiers in (approximate/continuous/compact) it starts to sound more like math and less like a miracle. Incidentally, one thing of great interest is, how does the number of hidden units required behave as a function of dimensionality of the input domain. In dramatic language, "Can neural networks get around the curse…
I agree. The problem with attempting to make subtle technical points digestible by untrained people is that the larger meaning of the exercise is likely to be lost along with the details. This would probably be a good demonstration to show to investors in your NN-based startup, because it seems impressive. But if I had to state the take-home message of this that the average person should care about, I'm not sure there is one. There are simpler systems with the same property that are taught to undergraduate engineers (and NNs don't represent a novel path to having it, even); there's a great distance between approximating functions and solving practical ML problems.
Edit: To me, personally, I would have been more interested had this been an argument about back propagation for training NNs. I suspected NNs could do this because of what's in them, but training is as much of a constraint on what they can actually do.
Earlier quoted context omitted.
> how does the number of hidden units required behave as a function of dimensionality of the input domain If I recall correctly, a non-linear problem can be solved as a linear problem if you consider more dimensions. The hidden layer add dimensions. So, it's not a function of the input domain but of the problem domain, which usually isn't explicitly known.
The question in the OP concerns approximation of the f(.) in: y = f(x) where the input "x" is d-dimensional (say). Some problems (i.e., choice of "f") could be easy. Maybe f only depends on one element of x, for example. There would be no curse of dimensionality in this case. Same situation if "f" depends only on any fixed number of elements of "x". I think this is roughly what you mean by "dimension of problem domai…
Earlier quoted context omitted.
You should explain in concrete terms why that is the case. I think its apparent that the human brain is a much more complex and advanced neural network (Intel 4004 vs Intel i7 perhaps?), but to say it is not is interesting and I would like to hear why.
Because the neural network described there is just an abstraction that has mathematical and (with some modification) practical utility. Real neurons do not behave in the way the model behaves. Synapses have plasticity, not just neurons. I'm not sure if synaptic efficacy (i.e. how much pre-synaptic input influences output) is fully understood even now. To put it simply, it is possible for ANNs to fail at something and…
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
NNs themselves are sort of agnostic to the algorithm you use to set their weights. It's sort of like saying "computers are Turing complete, but can we ever prove that human programmers are smart enough to program any function."
http://serialprog.blogspot.ca/2014/07/neural-networks-like-c...
My point of view was that neurons in neural nets are essentially analogue logic gates. Given that combinations of logic gates are Turing complete, combinations of neural net neurons should be also.
My writing is not quite as nice or rigorous as the parent post, but the whole point of the blog is to get better at self-expression and explaining things.
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
comparing neural net neurons to logic gates in a computer circuit or FPGA, gives a nice intuitive understanding of why the training works in the first place. The training essentially sets up a custom virtual circuit for whatever function you need.
Given this property, it seems intuitive that they would be able to learn any function, since any function is representable as logic gates.
Who cares if they can compute any function. The important question is, can they learn any function, and can they learn in a way that can generalize? (And clearly they can for many useful domains).
The initial flurry of research activity slowed to a background simmer, however. As a steady march of implementation problems was knocked down, it became clear that the limitations of neural nets were inherent to neural nets themselves, rather than any particular implementation detail. For example, the Lagrangians calculated by SVMs are provably convex, in contrast to the lack of a convexity guarantee for neural nets, which happily gradient descent into local minima. There were workarounds, of course, with regularization techniques, stochastic descent, etc. providing some measure of relief, but a migration of research interest away from neural nets seemed increasingly promising, and today, the migration seems largely complete.
Earlier quoted context omitted.
You also need the qualifier "...any continuous function, on a compact set. " Once you add all three qualifiers in (approximate/continuous/compact) it starts to sound more like math and less like a miracle. Incidentally, one thing of great interest is, how does the number of hidden units required behave as a function of dimensionality of the input domain. In dramatic language, "Can neural networks get around the curse…
>> Once you add all three qualifiers in (approximate/continuous/compact) it starts to sound more like math and less like a miracle. I agree. The problem with attempting to make subtle technical points digestible by untrained people is that the larger meaning of the exercise is likely to be lost along with the details. This would probably be a good demonstration to show to investors in your NN-based startup, because i…
Later on in the article, it's qualified, but tempers are already rising.
Here come people with their non-computable functions, their unmeasurable functions, their nowhere-continuous functions, all wanting to get approximated. In sup norm! On unbounded sets!
Irksome, greedy, and vexatious.