Nice write up. I gave a presentation on DBNs for my Neural Networks class in Fall 2011. If you'd like references to the relevant papers and some more details on the algorithms and applications, here are the slides: https://docs.google.com/presentation/d/18vJ2mOmb-Cbqsk0aNoUM...
Resurgence of Neural Networks
21–30 of 62 posts
Re: Resurgence of Neural Networks
#22Really interesting stuff. I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback. One of the performanc…
The problem with stuff like this is 16Mhz * 100 is far less than 3500Mhz * 4. So doing this in software on the desktop is generally a much better idea let alone 750Mhz * 1000+ if you can get a good GPU implementation. You also hit significant speed of light and bandwidth issues if you want to network a lot of these together because neurons don't just talk to there 4 closest friends. PS: Still a fun project, just hard…
It's no doubt difficult to scale, but there is no hobby-scale project that I'm aware of doing something like this. A quad-core Beast-PU from Intel or AMD is great if you're looking to get work done, but terrible if you're looking to open the hood and poke at the bits and innards. The most I've seen lately has been someone loading an existing cluster OS on 64 Raspberry Pis calling it a day. There might even be some interesting considerations for power efficiency and algorithms for disabling and reenabling nodes in the graph. There might even be some insights learned on improvements and/or exploits for other processor network systems that currently exist, like vehicle CANbuses.
Re: Resurgence of Neural Networks
#23Really interesting stuff. I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback. One of the performanc…
You may want this: http://www.greenarraychips.com/index.html
That's pretty neat. Thanks for the link, I know a few other people who would be interested in this, too.
Re: Resurgence of Neural Networks
#24First, it's Geoffrey, not Gregory Hinton. Here's a very good tech talk from him about RBMs: http://www.youtube.com/watch?v=AyzOUbkUf3M That said, both approaches loosely mirror the function of the brain, as neurons are not simple threshold devices, and both backpropagation and the RBMs training algorithms do not have a biophysical equivalent.
Oh sorry. I fixed it. Sorry Geoffrey!
Second, it's Geoffrey, not Gregory Hinton.
Re: Resurgence of Neural Networks
#25Really interesting stuff. I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback. One of the performanc…
I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). There are many ways to avoid this, for example have a look at: http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method In tradition…
Both RProp and Conjugate Gradient are descent methods-- they find a path to a local minima from the starting configuration. They do not help with finding the global minimum.
Using simulated annealing to guide random initializations can help find a better minima, but to get the global minima with simulated annealing takes an inordinate amount of time.
Re: Resurgence of Neural Networks
#26That Coursera class has been showing a start date of "Oct 1st 2012" for a while. Does anyone know when the next class might be?
Re: Resurgence of Neural Networks
#27[deleted]
Re: Resurgence of Neural Networks
#28That Coursera class has been showing a start date of "Oct 1st 2012" for a while. Does anyone know when the next class might be?
Haven't heard anything about the next class starting, but I believe you can still sign up and download all the videos.
Re: Resurgence of Neural Networks
#29I've been working for several years as the "applications developer" for a neural net lab. The neural lab has spent 11 years developing and refining a neural net pipeline - a series of neural nets which given one or more photos of a person's face, the pipeline performs forensically accurate 3D Reconstructions of the person's face and head. The system is used by government & police agencies the world over when trying t…
Thats some pretty impressive stuff. I would suggest changing your site name though, it seems a bit spamish (I almost didn't go for that reason).
Re: Resurgence of Neural Networks
#30Earlier quoted context omitted.
I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). There are many ways to avoid this, for example have a look at: http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method In tradition…
Both RProp and Conjugate Gradient are descent methods-- they find a path to a local minima from the starting configuration. They do not help with finding the global minimum. Using simulated annealing to guide random initializations can help find a better minima, but to get the global minima with simulated annealing takes an inordinate amount of time.
(Poor) local minima can be sometimes reached in back propagation due to a bad setting of the learning rate and RProp does help with that. You can also repeat the learning process using one of those algorithms many times with different initial random weights and use the results of the best attempt. I think this most often suffices in practical applications of NN for getting decent results, you are right I got some of this wrong, I basically wanted to say that backprop is no longer state of the art in training and that optimization is the relatively easy part of learning.