Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Deep Learning 101

markus.com

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Deep Learning 101

#41

Post by nrmn »

rahimiali wrote: post #30
rm999 wrote: post #23

Earlier quoted context omitted.

The history of AI is really interesting. Perceptrons were extremely oversold by their inventor, Frank Rosenblatt after he introduced then in 1958. This led to a lot of funding and interest in AI and perceptrons. Then, in 1969, Marvin Minsky coauthored a book Perceptrons which harshly criticized how underpowered perceptrons were. Most famously, the book proved that a perceptron could not model a simple XOR function. I…

Signal processing mysticism repeats itself every 20 years and has been fueled by tremendous hype since its debut 400 years ago: 1. Linear Regression (which, admittedly, was amazing) 2. Fourier Analysis (which is linear regression on orthonormal bases of functions. it blew people's minds) 3. Perceptrons (which is linear regression but with a logistic loss. it went back to its old name of "logistic regression" once its…

Could you explain the filtering "their inputs through fixed or random nonlinearities"? I haven't heard of this before.

Re: Deep Learning 101

#42

Post by dwiel »

ma2rten wrote: post #34

I've followed the developments in Neural Networks somewhat, but have never applied deep learning so far. This is seems like a good place to ask a couple of question I've been having for a while. 1. When does it make sense to apply deep learning? Could it potentially be applied successful applied to any difficult problem given enough data? Could it also be good at the type of problems that Random Forest, Gradient Boos…

I don't have great answers to the other questions, though I too am interested in them.

#5) [1] has a some python code and timings mixed in to the docs. One such example (stacked denoising autoencoders on MNIST):

    By default the code runs 15 pre-training epochs for each layer,             
    with a batch size of 1. The corruption level forthe first layer is          
    0.1, for the second 0.2 and 0.3 for the third. The pretraining              
    learning rate is was 0.001 and the finetuning learning rate is              
    0.1. Pre-training takes 585.01 minutes, with an average of 13               
    minutes per epoch. Fine-tuning is completed after 36 epochs in              
    444.2 minutes, with an average of 12.34 minutes per epoch. The              
    final validation score is 1.39% with a testing score of                     
    1.3%. These results were obtained on a machine with an Intel Xeon           
    E5430 @ 2.66GHz CPU, with a single-threaded GotoBLAS.
#6) The size of the NN is not typically num_features * num_classes, but rather num_features * num_layers where num_layers is commonly 3-10 or so. If you want a (multi-class) classifier, you first feed your neural network a bunch of examples, unsupervised. Then once you've got your NN built, you feed the outputs of the NN to a classifier like SVM or SGD. The idea is that the net provides more meaningful features than you would have if you used hand crafted features or the raw input data itself.

[1] http://deeplearning.net/tutorial/SdA.html#sda

Re: Deep Learning 101

#43

Post by rahimiali »

nrmn wrote: post #41
rahimiali wrote: post #30

Earlier quoted context omitted.

Signal processing mysticism repeats itself every 20 years and has been fueled by tremendous hype since its debut 400 years ago: 1. Linear Regression (which, admittedly, was amazing) 2. Fourier Analysis (which is linear regression on orthonormal bases of functions. it blew people's minds) 3. Perceptrons (which is linear regression but with a logistic loss. it went back to its old name of "logistic regression" once its…

Could you explain the filtering "their inputs through fixed or random nonlinearities"? I haven't heard of this before.

you've actually probably done this yourself. it's often called "featurization". for example, instead of applying a linear learner on vectors x in R^d, you apply it to vectors f(x), where f computes a bunch of features on x. a popular choice for f are the d-th order monomials. hashing families are another good idea (Alex Smola does this). more generally, any random nonlinear function f is a good candidate (i call that analysis "Random Kitchen Sinks"). when x is structured data, f usually just returns counts in histogram bins of some kind.

Re: Deep Learning 101

#44

Post by faxmulder »

mbeissinger wrote: post #38
faxmulder wrote: post #37

Very interesting stuff written in a clear way. I'm actually finishing my master thesis on music genre recognition through machine learning, which is focused more on traditional ensemble learning, but I think that it would be nice to study deep learning in greater detail. Thanks!

Awesome, do you have any demos for the music genre recognition?

not yet, I've still some work to do. One question: do you think that Optimum-Path Forests could be used also in the context of deep learning?

Re: Deep Learning 101

#45

Post by kkjkok »

jules wrote: post #33
brandonb wrote: post #18

Earlier quoted context omitted.

That's what I thought too! But according to my friends on the Google Brain team, unsupervised pretraining is now thought to be an irrelevant detour. In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima. The theory was that you'd pretrain to find a good initial set of connection weights, then apply backprop to "fine-tune…

> And around ~2012, a bunch of researchers have reported you don't even need 2nd-derivative information. You just have to initialize the neural net properly. This sounds very interesting. How do you property initialize the weights? Do you have a link to a paper about this?

Check out this paper:

Practical recommendations for gradient-based training of deep architectures, Y. Bengio

http://arxiv.org/abs/1206.5533

There is a section on weight initialization on page 15. In general, this paper has a lot of good information in one place.

Re: Deep Learning 101

#46

Post by ma2rten »

dwiel wrote: post #42
ma2rten wrote: post #34

I've followed the developments in Neural Networks somewhat, but have never applied deep learning so far. This is seems like a good place to ask a couple of question I've been having for a while. 1. When does it make sense to apply deep learning? Could it potentially be applied successful applied to any difficult problem given enough data? Could it also be good at the type of problems that Random Forest, Gradient Boos…

I don't have great answers to the other questions, though I too am interested in them. #5) [1] has a some python code and timings mixed in to the docs. One such example (stacked denoising autoencoders on MNIST): By default the code runs 15 pre-training epochs for each layer, with a batch size of 1. The corruption level forthe first layer is 0.1, for the second 0.2 and 0.3 for the third. The pretraining learning rate…

I understand that this unsupervised approach is out of fashion already.

https://plus.google.com/+YannLeCunPhD/posts/UVT2fYTfoAC

Post reply on HN