Deep Learning Is Not So Mysterious or Different
1–10 of 139 posts
Re: Deep Learning Is Not So Mysterious or Different
#2Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12)
Re: Deep Learning Is Not So Mysterious or Different
#3Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036
I'm a huge fan of HN just for replies such as this that smash the OP's post/product with something better. It's like at least half the reason I stick around here.
Thanks for the great read.
Re: Deep Learning Is Not So Mysterious or Different
#4[flagged]
Re: Deep Learning Is Not So Mysterious or Different
#5Anyone who wants to demystify ML should read: The StatQuest Illustrated Guide to Machine Learning [0] By Josh Starmer.
To this day I haven't found a teacher who could express complex ideas as clearly and concisely as Starmer does. It's written in an almost children's book like format that is very easy to read and understand. He also just published a book on NN that is just as good. Highly recommend even if you are already an expert as it will give you great ways to teach and communicate complex ideas in ML.
[0]: https://www.goodreads.com/book/show/75622146-the-statquest-i...
Re: Deep Learning Is Not So Mysterious or Different
#6> rather than restricting the hypothesis space to avoid overfitting, embrace a flexible hypothesis space, with a soft preference for simpler solutions that are consistent with the data. This principle can be encoded in many model classes, and thus deep learning is not as mysterious or different from other model classes as it might seem.
How does deep learning do this? The last time I was deeply involved in machine learning, we used a penalized likelihood approach. To find a good model for data, you would optimize a cost function over model space, and the cost function was the sum of two terms: one quantifying the difference between model predictions and data, and the other quantifying the model's complexity. This framework encodes exactly a "soft preference for simpler solutions that are consistent with the data", but is that how deep learning works? I had the impression that the way complexity is penalized in deep learning was more complex, less straightforward.
Re: Deep Learning Is Not So Mysterious or Different
#7[flagged]
It's a higher quality article than half the submissions to NeuroIPS and the rest of the AI/ML conferences, because it has potential to remain relevant next year, and because of its high didactic content.
I wouldn't classify it as a "hot take".
Re: Deep Learning Is Not So Mysterious or Different
#8An example, which is interesting, in which "deep" networks are necessary, is discussed in this fascinating and popular recent paper on RNNs [1]. Despite the fact that the minGRU and minLSTM models they propose don't explicitly model ordered state dependency, they can learn them as long as they are deep enough (deep >= 3):
> Instead of explicitly modelling dependencies on previous states to capture long-range dependencies, these kinds of recurrent models can learn them by stacking multiple layers.
Re: Deep Learning Is Not So Mysterious or Different
#9Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036
Good read, thanks for sharing
Re: Deep Learning Is Not So Mysterious or Different
#10Agreed, but PAC-Bayes or other descendants of VC theory is probably not the best explanation. The notion of algorithmic stability provides a (much) more compelling explanation. See [1] (particularly Sections 11 and 12) [1] https://arxiv.org/abs/2203.10036
I'm a huge fan of HN just for replies such as this that smash the OP's post/product with something better. It's like at least half the reason I stick around here. Thanks for the great read.
>smash with something better
Not a fan of the aggressive rhetoric here...