Sitting in a class right now reading this while Hinton is teaching neural nets.
Resurgence of Neural Networks
31–40 of 62 posts
Re: Resurgence of Neural Networks
#32Nice write up. I gave a presentation on DBNs for my Neural Networks class in Fall 2011. If you'd like references to the relevant papers and some more details on the algorithms and applications, here are the slides: https://docs.google.com/presentation/d/18vJ2mOmb-Cbqsk0aNoUM...
This was a really fun read-thank you! If you don't mind answering, I've got a few questions:
1. "Vanishing gradients after 2-3 layers"-does this mean that the partial derivatives tend to be smaller on the higher layers, and therefore the network finds local minima that aren't very useful?
2. Step 3 (p 18) mentions that the outputs are not continuous variables, they're binary. What's the reasoning behind that?
Re: Resurgence of Neural Networks
#33Neural networks are interesting, but they are not a simulation of a brain. Brains re-wire their connections, these "neural networks" cannot.
Re: Resurgence of Neural Networks
#34Nice write up. I gave a presentation on DBNs for my Neural Networks class in Fall 2011. If you'd like references to the relevant papers and some more details on the algorithms and applications, here are the slides: https://docs.google.com/presentation/d/18vJ2mOmb-Cbqsk0aNoUM...
This was a really fun read-thank you! If you don't mind answering, I've got a few questions: 1. "Vanishing gradients after 2-3 layers"-does this mean that the partial derivatives tend to be smaller on the higher layers, and therefore the network finds local minima that aren't very useful? 2. Step 3 (p 18) mentions that the outputs are not continuous variables, they're binary. What's the reasoning behind that?
1. Basically. It means that the network has a hard time pulling itself in any direction since the weights in the deeper layers are never really adjusted by very much.
2. It's been a while since I read the paper, but I believe that the justification has to do with the proof of convergence of Gibbs sampling. I haven't tried using continuous values, so I can't give an intuition for what happens in those cases.
Re: Resurgence of Neural Networks
#35Sitting in a class right now reading this while Hinton is teaching neural nets.
I can remember times being a student, learning a lot, but losing focus of how amazing it was I had dedicated time solely to study. As someone working full-time now, who only gets to learn new fascinating math in his precious free time, I implore you to fully embrace the awesome opportunity in front of you. Enjoy the lecture, and milk whatever you can from this man's teaching.
Re: Resurgence of Neural Networks
#36I'm the President and Chief Scientist of Kaggle, which ran the drug discovery project mentioned in the article. As it happens, I did my Strata talk on Tuesday about just this topic. I will be repeating the talk in webcast form (for free) in a few weeks: http://oreillynet.com/pub/e/2538 . I'll be focussing more on the data science implications, rather than implementation details.
Re: Resurgence of Neural Networks
#37Really interesting stuff. I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). This RBM approach sounds better than back-propagation, but it also sounds like it would be prone to runaway feedback. One of the performanc…
I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). There are many ways to avoid this, for example have a look at: http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method In tradition…
Well, if you've got 20 numbers to multiply you'll get the job done fastest if you do them in parallel with 20 dedicated multipliers.
There's an obvious vision of building a "neural circuit" where there is some specialized processor for each neuron but my guess is that it gets difficult when you consider the communication fabric required between the layers.
Re: Resurgence of Neural Networks
#38Earlier quoted context omitted.
I had once attempted to build a genetic algorithm for manipulating the synapse weights, specifically because of the problems of traditional back-propagation falling into local minima (unfortunately, some serious shit at work made it drop by the wayside). There are many ways to avoid this, for example have a look at: http://en.wikipedia.org/wiki/Rprop http://en.wikipedia.org/wiki/Conjugate_gradient_method In tradition…
Well, if you've got 20 numbers to multiply you'll get the job done fastest if you do them in parallel with 20 dedicated multipliers. There's an obvious vision of building a "neural circuit" where there is some specialized processor for each neuron but my guess is that it gets difficult when you consider the communication fabric required between the layers.
The first artificial neural networks were actually done in hardware, here is Frank Rosenblatt with his Perceptron:
http://www.enzyklopaedie-der-wirtschaftsinformatik.de/wi-enz...
As said, it is an interesting thing to consider as an alternative computer architecture and what not, but I just have some doubts if people practically using neural networks right now really run into performance problems because of having too little cores. I don't think this is more true for neural networks than for anything else.
Re: Resurgence of Neural Networks
#39As mentioned in this thread by nicholasjarnold, Jeff Hawkins work on HTM (detailed in his excellent book "On Intelligence") seems superficially similar to this. Has anyone had experience of both approaches. HTM seems to have much more structure in the network, but I know next to nothing about AI and would love to hear from those who know a bit more.
I have some experience with both, but can't give a great comparison. The tldr is I've always had a better impression of Hinton than Hawkins, and have studied/followed Hinton's approaches much more carefully.
In late 2006/early 2007 I was working a lot with standard two layer feed-forward neural networks (first for my research and then for my job). Hinton had a great paper on practical deep networks at NIPS 2006 (a big AI/machine learning conference), which sparked my interest in more complex neural networks. I had read Hawkins' book a few years earlier, and my impression of it was somewhat negative; I thought it was a really interesting book, but it was too fluffy and high-level to be intriguing. He hit a lot of points about hierarchies in intelligence that were intriguing but not new or drastically insightful. After NIPS I downloaded some of Numenta's code (numenta is Hawkins' company) and it was pretty slow on toy problems so I didn't spend too much time with it - this isn't a judgement of their code, I just didn't have the time to dig deeply into it. My impression at the time, which may be unfair, is that Numenta's approach was ad-hoc while Hinton's was principled. I was negatively biased by Hawkins' book and my professors' opinions of him vs Hinton.
Re: Resurgence of Neural Networks
#40I remember running into Hinton one afternoon back in 2005 while on St. George. He was walking home, and especially cheerful from having just figured out how to do learning efficiently on deep belief nets. It's amazing to see the influence this work has had.