31–40 of 46 posts
What does it take to be good at machine learning such as this? In terms of mathematics, and computer science knowledge? I know how to code through self learning, and I've pretty much solely done web development. So I barely know much CS. Also not very good at math. So what are the essential prerequisites you would say are necessary for doing neat, useful stuff with machine learning?
Earlier quoted context omitted.
The history of AI is really interesting. Perceptrons were extremely oversold by their inventor, Frank Rosenblatt after he introduced then in 1958. This led to a lot of funding and interest in AI and perceptrons. Then, in 1969, Marvin Minsky coauthored a book Perceptrons which harshly criticized how underpowered perceptrons were. Most famously, the book proved that a perceptron could not model a simple XOR function. I…
Signal processing mysticism repeats itself every 20 years and has been fueled by tremendous hype since its debut 400 years ago: 1. Linear Regression (which, admittedly, was amazing) 2. Fourier Analysis (which is linear regression on orthonormal bases of functions. it blew people's minds) 3. Perceptrons (which is linear regression but with a logistic loss. it went back to its old name of "logistic regression" once its…
I would say that the next bing thing is more: Realizing even more that Neural Nets is an optimization problem, and instead of using some heuristics, wait for some Russian mathematician to derive the right SGD schedule / batch solver for the problem. Then what the 1,000 of Google computers have been able to do for the cat face detector, we'll be able to do it on a smartphone chip. People have to realize that Deep Learning is a bit of a "brute force" solution for the moment (each node is a linear model). We need to derive smarter algorithms.
Earlier quoted context omitted.
Yep the ideas from the 50's have definitely reappeared now we have the compute power and methods to implement them at a large scale. That article gives a nice perspective. One of the best breakthroughs has been this notion of layer-wise pretraining, which allows the backpropagation algorithm to not get stuck in local minima so easily. It provides a good guess to the starting starting points for the weights. Otherwise…
That's what I thought too! But according to my friends on the Google Brain team, unsupervised pretraining is now thought to be an irrelevant detour. In 2006, Hinton introduced greedy layer-wise pretraining, which was intended to solve the problem of backpropagation getting stuck in poor local optima. The theory was that you'd pretrain to find a good initial set of connection weights, then apply backprop to "fine-tune…
This sounds very interesting. How do you property initialize the weights? Do you have a link to a paper about this?
1. When does it make sense to apply deep learning? Could it potentially be applied successful applied to any difficult problem given enough data? Could it also be good at the type of problems that Random Forest, Gradient Boosting Machines are traditionally good at versus the problems that SVMs are traditionally good at (Computer Vision, NLP)? [1]
2. How much data is enough?
3. What degree of tuning is required to make it work? Are we at the point yet where deep learning works more or less out the box?
4. Is it fair to say that dropout and maxout always work better in practice? [2]
5. What is the computational effort? How long e.g. does it take to classify an ImageNet image (on a CPU / GPU)? How long does it take train a model like that?
6. How on earth does this fit into memory? Say in ImageNet your have (256 pixels * 256 pixels) * (10,000 classes) * 4 bytes = 2.4 GB, for a NN without any hidden layers.
[1] I am overgeneralizing somewhat, I know. It's my way to avoid overfitting.
[2] My lunch today was free.
Personally, I've found that I don't retain much of this sort of material without working through exercises. If you learn the same way, you might want to check out the series of progressive exercises from Andrew Ng here: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial For reference, I have a copy of my solutions here: https://github.com/danluu/UFLDL-tutorial . Debugging broken learning algorithms can be tediou…
Personally, I've found that I don't retain much of this sort of material without working through exercises. If you learn the same way, you might want to check out the series of progressive exercises from Andrew Ng here: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial For reference, I have a copy of my solutions here: https://github.com/danluu/UFLDL-tutorial . Debugging broken learning algorithms can be tediou…
Very interesting stuff written in a clear way. I'm actually finishing my master thesis on music genre recognition through machine learning, which is focused more on traditional ensemble learning, but I think that it would be nice to study deep learning in greater detail. Thanks!
Personally, I've found that I don't retain much of this sort of material without working through exercises. If you learn the same way, you might want to check out the series of progressive exercises from Andrew Ng here: http://ufldl.stanford.edu/wiki/index.php/UFLDL_Tutorial For reference, I have a copy of my solutions here: https://github.com/danluu/UFLDL-tutorial . Debugging broken learning algorithms can be tediou…
For some more elementary material, I also recommend Andrew Ng's machine learning course on Coursera. He's a great teacher.