It looks like the article was deleted ("Oops! That page can’t be found"). Here is the Google Cache:
https://webcache.googleusercontent.com/search?q=cache:5HMZ_Z...
Why deep learning works even though it shouldn’t
141–148 of 148 posts
Re: Why deep learning works even though it shouldn’t
#142Earlier quoted context omitted.
Other fields have their popular message boards. It's not just programmers.
What are the popular message boards for practicing physicists and mathematicians?
The Mathematics Stack Exchange has consistently surprised me (in a good way) in their helpfulness and rigor.
Re: Why deep learning works even though it shouldn’t
#143Setting aside the primary subject, this is an excellent observation: > What I find however is that there are a base of unspoken intuitions that underlie expert understanding of a field, that are never directly stated in the literature, because they can’t be easily proved with the rigor that the literature demands. And as a result, the insights exist only in conversation and subtext, which make them inaccessible to th…
While most people think this "knowledge" should be organized and even shared, I strongly disagree. For context, I have worked in large research labs, ML engineering organizations and startups and have encountered many people across the engineer and research spectrum. These intuitions are often wrong and arise due to the lack of vocabulary in correctly describing the mechanisms that occur. From a researcher's standpoi…
This is a very good point. Especially in ML, there have been many cases where very smart people were wrong with their intuitions (the original explanation of batchnorm comes to mind), or maybe their intuitions were correct, but attempts to explain the intuition (especially when there's a race to publish) led to wrong conclusions. I still think intuitions should be discussed and shared, but with a clearly stated caveat like: "that's just my guess, we don't know what's really happening there". This is how I usually explain to others (and to myself) my experimental results.
Re: Why deep learning works even though it shouldn’t
#144I don't think we can say for sure that early stopping is the main reason deep networks generalize. Double descent [1] shows that models continue to improve even once they've "interpolated" the training data (fit every point perfectly), and critical periods [2] suggest that the early part of training is responsible for most of the generalization performance even though much of the numerical improvement happens later.…
I think you misunderstood the point of deep double descent. The x-axis is not number of training epochs, it is model capacity.
I think you'd be interested in https://arxiv.org/abs/1611.03530. It discusses how SGD is an implicit regularizer. We also actually want high variance weights for symmetry breaking.
Re: Why deep learning works even though it shouldn’t
#145The article was gone?
Re: Why deep learning works even though it shouldn’t
#146Earlier quoted context omitted.
That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850
They are both universal approximators. So are support vector machines, gaussian processes and gradient boosted trees. Yet the performance of neural networks is unrivaled in certain tasks, as has been proven over and over again. As a whole, that paper is quite bad (and still unpublished, probably blocked by peer review) because (1) it only considers fully connected networks (which are a minority of models used nowaday…
To be fair, most of the machine learning literature had no or a poor excuse for peer review. And many deep learning layers can be described by dense layers. For example, convolution.
You almost certainly can frame Go playing as polynomial regression but there would undoubtedly be numerical precision & other gradient issues. Deep learning is a practice is remarkably effective, no disagreement there.
Re: Why deep learning works even though it shouldn’t
#147Earlier quoted context omitted.
That's a strong unsubstantiated claim. On the other hand, there's been some nice theoretical work arguing that deep learning is a form of polynomial regression. https://arxiv.org/abs/1806.06850
Let us know when polynomial regression succeeds at any machine learning task. A lot of people publish results that say deep learning is "just" something else, where the something else doesn't work.
How about 80% accuracy on CIFAR-10 with unsupervised training?
=> logistic regression + Kmeans
http://ai.stanford.edu/~acoates/papers/coatesleeng_aistats_2...
Re: Why deep learning works even though it shouldn’t
#148Another interesting explanation for deep learning's success in the physical world:
Why does deep and cheap learning work so well?
Max Tegmark et al.,