Resurgence of Neural Networks

> Now, when I say Artificial Intelligence I’m really only referring to Neural Networks. There are many other kinds of A.I. out there (e.g. Expert Systems, Classifiers and the like) but none of those store information like our brain does (between connections across billions of neurons).

This is a middle-brow dismissal of almost the entire field of A.I. because it does not meet an unnecessarily narrow restriction. (Which, by the way, neural nets don't either. Real neurons are analog-in, digital-out, stochastic processes with behavior influenced by neural chemistry and with physical interconnectivity and timing among other things not accurately modeled at all by any neural net. It's closer modeling to the mechanisms of the brain, but far from equivalent and as a CogSci student you should know that.)

A.I. is the science of building artifacts exhibiting intelligent behavior, intelligence being loosely defined as what human minds do. But in theory and in practice, what human minds do is not the same thing as how they do it.

The human mind does appear to be a pattern matching engine, with components that might indeed be well described as a hidden Markov model or restricted Boltzmann machine. It may be that our brains are nothing more than an amalgamation of some 300 million or so interconnected hidden Markov models. That's Ray Kurzweil's view in How to Create a Mind, at any rate.

However it is a logical fallacy to infer that neural nets are the only or even the best mechanism for implementing all aspects of human-level intelligence. It's merely the first thing evolution was able to come up with through trial and error.

Take the classical opposite of neural nets, for example: symbolic logic. If given a suitable base of facts to work from and appropriate goals, a theorem prover on your cell phone could derive all mathematics known up to the early 20th century (and perhaps beyond), without the possibility of making a single mistake. And do it on a fraction of the energy you spend splitting a bill and calculating tip. A theorem prover alone does not solve the problem of initial learning of ontologies or reasoning about uncertainty in a partially observable and even sometimes inconsistent world. But analyzing memories and perception for new knowledge is a large part of what human minds do (consciously, at least), and if you have a better tool, why not use it?

Now I myself am enamored by Hilton-like RBM nets. This sort of unassisted deep-learning is probably a cornerstone in creating a general process for extracting any information from any environment, a central task of artificial general intelligence. However compared with specialized alternatives, neural nets are hideously inefficient for many things. Doesn't it make sense then to use an amalgam of specialized techniques when applicable, and fall back on neural nets for unstructured learning and other non-specialized tasks? Indeed this integrative approach is taken by OpenCog, although they plan to use DeSTIN deep-learning instead of Hilton-esque RBM's, in part because the output of DeSTIN is supposedly more easily stored in their knowledge base and parsed by their symbolic logic engine.

After all, a deep belief network starts with an RBM for unsupervised pre-training, but the finetuning stage that follows just treats the network as a standard MLP using backprop.

Also, you can use an autoencoder instead of an RBM, which I think are getting better results these days? And there are better regularization techniques for backprop now--weight decay, momentum, L1/L2 regularization, dropout, probably more that I'm leaving out.

The pre-training (RBM or autoencoder) helps to not get stuck in local minimas, but there's also interesting research that suggests you're not even getting stuck in local minima so much as you're getting stuck in these low slope, high curvature corridors that gradient descent is blind to, so people are looking into second order methods that can take curvature into account so you can take big steps through these canyons and smaller steps when things are a bit steeper. Or something like that :-)

All that being said, anyone care to weigh in on the pros/cons of RBMs vs something like a contractive autoencoder? No such thing as a free lunch, so what are the key selling points of RBMs at this point? I keep seeing them pop up, but afaik, they don't provide a particular advantage over autoencoder variants.

Great article though, I'm really glad to see more and more people getting interested in neural networks, they've come a long way and people are just starting to wake up to that.

(I wrote it... :-)