Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning 101

markus.com

Post reply on HN
Print view View original

11–20 of 46 posts

Re: Deep Learning 101

#11

Post by mbeissinger »

Noxchi wrote: post #9

What does it take to be good at machine learning such as this? In terms of mathematics, and computer science knowledge? I know how to code through self learning, and I've pretty much solely done web development. So I barely know much CS. Also not very good at math. So what are the essential prerequisites you would say are necessary for doing neat, useful stuff with machine learning?

Definitely a solid foundation in linear algebra and statistics (mostly Bayesian) are necessary for understanding how the algorithms work. Check out the wiki portals (http://en.wikipedia.org/wiki/Machine_learning) and (http://en.wikipedia.org/wiki/Artificial_intelligence) for overviews of the most common approaches.

Also, Andrew Ng's coursera course on machine learning is amazing (https://www.coursera.org/course/ml) as well as Norvig and Thrun's Udacity course on AI (https://www.udacity.com/course/cs271)

Re: Deep Learning 101

#12

Post by seiji »

brandonb wrote: post #10

This is a cool tutorial! It's ironic that deep neural networks have become the biggest machine learning breakthrough of 2013: they were also the biggest machine learning breakthrough of 1957. The idea dates back to the Perceptron, one of the oldest ideas in AI. One thing to note: although there was a lot of initial excitement about Restricted Boltzman Machines, Auto-encoders, and other unsupervised approaches, the be…

RBMs and auto-encoders use backprop too. They just use it for "fine tuning" (due to running pre-training first) instead of propagating error derivatives from randomly initialized weights.

(Thus concludes the smartest thing I've said all day.)

Re: Deep Learning 101

#13

Post by mbeissinger »

brandonb wrote: post #10

This is a cool tutorial! It's ironic that deep neural networks have become the biggest machine learning breakthrough of 2013: they were also the biggest machine learning breakthrough of 1957. The idea dates back to the Perceptron, one of the oldest ideas in AI. One thing to note: although there was a lot of initial excitement about Restricted Boltzman Machines, Auto-encoders, and other unsupervised approaches, the be…

Yep the ideas from the 50's have definitely reappeared now we have the compute power and methods to implement them at a large scale. That article gives a nice perspective.

One of the best breakthroughs has been this notion of layer-wise pretraining, which allows the backpropagation algorithm to not get stuck in local minima so easily. It provides a good guess to the starting starting points for the weights. Otherwise, the biggest issue with backpropagation historically has been the diffusion of weights as the layers increase; it is hard to attribute the causality or what portion of the update weighting should be applied to each node since it grows exponentially. This pretraining idea helps against that.

Re: Deep Learning 101

#14

Post by shon »

Google, Twitter, Netflix, Yelp, Pandora and more are speaking on Deep Learning and RecSys this Friday at MLconf in San Francisco. We're trying to get a streaming solution going as well for those who can't make it. http://mlconf.com

DISCLAIMER: This is my event

Re: Deep Learning 101

#15

Post by ingrownpsyche »

cocoflunchy wrote: post #3

OT but the text selection behavior on this page is fascinating! (Or horrific if you don't want to be nice). I've never seen anything like it. https://www.dropbox.com/s/4k72g8b2tl3mgzt/Screenshot%202013-...

That's fine for me, but I'm getting ligatures for every st. I thought it was deliberate (and a little pretentious really) but you're not getting them so hurray webfont or something I suppose.

Re: Deep Learning 101

#16

Post by mbeissinger »

shon wrote: post #14

Google, Twitter, Netflix, Yelp, Pandora and more are speaking on Deep Learning and RecSys this Friday at MLconf in San Francisco. We're trying to get a streaming solution going as well for those who can't make it. http://mlconf.com DISCLAIMER: This is my event

I'll definitely check this out if you get a stream going.

Re: Deep Learning 101

#17

Post by luu »

Personally, I've found that I don't retain much of this sort of material without working through exercises. If you learn the same way, you might want to check out the series of progressive exercises from Andrew Ng here: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial

For reference, I have a copy of my solutions here: https://github.com/danluu/UFLDL-tutorial. Debugging broken learning algorithms can be tedious in a way that's not particularly educational, so I tried to find a reference I could compare against when I was doing the exercises, and every copy I found had bugs. Hope having this reference helps someone.

Re: Deep Learning 101

#18

Post by brandonb »

mbeissinger wrote: post #13
brandonb wrote: post #10

This is a cool tutorial! It's ironic that deep neural networks have become the biggest machine learning breakthrough of 2013: they were also the biggest machine learning breakthrough of 1957. The idea dates back to the Perceptron, one of the oldest ideas in AI. One thing to note: although there was a lot of initial excitement about Restricted Boltzman Machines, Auto-encoders, and other unsupervised approaches, the be…

Yep the ideas from the 50's have definitely reappeared now we have the compute power and methods to implement them at a large scale. That article gives a nice perspective. One of the best breakthroughs has been this notion of layer-wise pretraining, which allows the backpropagation algorithm to not get stuck in local minima so easily. It provides a good guess to the starting starting points for the weights. Otherwise…

That's what I thought too! But according to my friends on the Google Brain team, unsupervised pretraining is now thought to be an irrelevant detour.

In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima. The theory was that you'd pretrain to find a good initial set of connection weights, then apply backprop to "fine-tune" discriminatively. And the theory seemed correct since the experimental results were good: http://www.cs.toronto.edu/~hinton/absps/fastnc.pdf http://machinelearning.wustl.edu/mlpapers/paper_files/NIPS20...

Does pretraining truly help solve the problem of poor local optima? In 2010, some empirical studies suggested the answer was yes: http://machinelearning.wustl.edu/mlpapers/paper_files/AISTAT...

But that same year, a student in Geoff Hinton's lab discovered that if you added information about the 2nd-derivatives of the loss function to backpropagation ("Hessian-free optimization"), you could skip pretraining and get the same or better results: http://machinelearning.wustl.edu/mlpapers/paper_files/icml20...

And around ~2012, a bunch of researchers have reported you don't even need 2nd-derivative information. You just have to initialize the neural net properly. Apparently, all the most recent results in speech recognition just use standard backpropagation with no unsupervised pretraining. (Although people are still trying more complex variants of unsupervised pretraining algorithms, often involving multiple types of layers in the neural network.)

So now, after seven years of work, we're back where we started: the plain ol' backpropgation algorithm from 1974 worked all along.

This whole topic is really interesting to me from a history of science perspective. What other old, discarded ideas from the past might be ripe, now that we have millions of times more data and computation?

Re: Deep Learning 101

#19

Post by mbeissinger »

brandonb wrote: post #18
mbeissinger wrote: post #13

Earlier quoted context omitted.

Yep the ideas from the 50's have definitely reappeared now we have the compute power and methods to implement them at a large scale. That article gives a nice perspective. One of the best breakthroughs has been this notion of layer-wise pretraining, which allows the backpropagation algorithm to not get stuck in local minima so easily. It provides a good guess to the starting starting points for the weights. Otherwise…

That's what I thought too! But according to my friends on the Google Brain team, unsupervised pretraining is now thought to be an irrelevant detour. In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima. The theory was that you'd pretrain to find a good initial set of connection weights, then apply backprop to "fine-tune…

Yes this is really interesting. I haven't read those other papers yet (definitely plan on it now thanks for the links), but Bengio's latest paper on denoising autoencoders from earlier this year (http://arxiv.org/abs/1305.6663) still used the unsupervised pretraining. Also the Theano implementation that I run experiments with uses it as well (but that code could be a year or two old).

Definitely going to be researching this more throughout the year.

Re: Deep Learning 101

#20

Post by shon »

mbeissinger wrote: post #16
shon wrote: post #14

Google, Twitter, Netflix, Yelp, Pandora and more are speaking on Deep Learning and RecSys this Friday at MLconf in San Francisco. We're trying to get a streaming solution going as well for those who can't make it. http://mlconf.com DISCLAIMER: This is my event

I'll definitely check this out if you get a stream going.

Check mlconf.com on Friday. The default will be Ustream here: http://www.ustream.tv/search?q=mlconf

We'll also post that and any updates to the main site on Friday.

Post reply on HN