Why does deep and cheap learning work so well?
1–10 of 53 posts
Re: Why does deep and cheap learning work so well?
#2Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics:
https://www.youtube.com/watch?v=5MdSE-N0bxs
The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.
Re: Why does deep and cheap learning work so well?
#3Man, it's good to see some actual information theory applied to deep learning for a change.
Re: Why does deep and cheap learning work so well?
#4Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics: https://www.youtube.com/watch?v=5MdSE-N0bxs The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.
Thank you!!! as a Physics PhD that was one of the first videos I found on deep learning, and having no idea what a big deal his insights were I promptly forgot who the speaker was (remembering only it was a name I knew intimately from my time in Physics). Have frequently gone back to try to find this unsuccessfully.
Re: Why does deep and cheap learning work so well?
#5Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics: https://www.youtube.com/watch?v=5MdSE-N0bxs The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.
[deleted]
Re: Why does deep and cheap learning work so well?
#6Man, it's good to see some actual information theory applied to deep learning for a change.
I'm sorry but this comment is just at odds with the current literature in deep learning.
A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see
the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'....
https://arxiv.org/search/?query=deep+learning+information+th...
Re: Why does deep and cheap learning work so well?
#7Man, it's good to see some actual information theory applied to deep learning for a change.
There's papers posted to arXiv every day using information theory to analyze deep learning...
Re: Why does deep and cheap learning work so well?
#8Same reason general relativity works. If you try to model results without a fundamental principle of the system's content's operation then you are going to have some limitations.
Re: Why does deep and cheap learning work so well?
#9Man, it's good to see some actual information theory applied to deep learning for a change.
I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'.... https://arxiv.org/search/?query=deep+learning+information+th...
I think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.
Re: Why does deep and cheap learning work so well?
#10Man, it's good to see some actual information theory applied to deep learning for a change.
I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'.... https://arxiv.org/search/?query=deep+learning+information+th...
Good. I hope to see more of them posted to HN.