Man, it's good to see some actual information theory applied to deep learning for a change.
[deleted]
Why does deep and cheap learning work so well?
11–20 of 53 posts
Re: Why does deep and cheap learning work so well?
#12Earlier quoted context omitted.
I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'.... https://arxiv.org/search/?query=deep+learning+information+th...
Good. I hope to see more of them posted to HN.
if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening...
but some of his key claims are probably contradicted by empirical results.
the original paper: https://arxiv.org/abs/1503.02406
a refutation:(with very harsh/unprofessional words by tishby in counter-refutation)
Re: Why does deep and cheap learning work so well?
#13Earlier quoted context omitted.
I'm sorry but this comment is just at odds with the current literature in deep learning. A few very prominent researchers have been trying to use information theoretic concepts to explain generalization for a while now-see the somewhat controversial branch Naftali Tishby developed...However many other papers apply 'actual information theory'.... https://arxiv.org/search/?query=deep+learning+information+th...
I think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.
trust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves
Re: Why does deep and cheap learning work so well?
#14This is a very nice paper that puts some "meaningful" names in neural network function. I like how eq. 8 neatly describes a neural network as a readout of an energy hamiltonian and an output distribution. They show that real-world data, thats is described by low-order polynomial hamiltonians , needs a small number of units and that "depth" gives the network its compositional/hierarchical ability. Even though some of the theory goes over my head, their main arguments seem to "fit" together very nicely.
So basically a deep network can "cheaply" (i.e. not fatally expensive) describe anything that occurs in nature, which is wonderful. I wonder however what will happen when we move to higher cognition and meta-cognition which requires the readout of network states that are not found in nature, but are generated internally. Would be interesting to know if we need much more brain or a little more. In any case very intersting read.
Re: Why does deep and cheap learning work so well?
#15A year ago, a group of us at UMD took a couple weeks to go through this paper. It's got some interesting insights, but like all theory papers right now, gaps remain. The construction they present for learning Hamiltonians of low polynomial order doesn't look a ton like any common production neural network modules, and the justification for why we would only be dealing with that Hamiltonian family in practice is unconvincing to me. That said, overall it's worth a close read. Section 2' parts A and B are the best summary of the connection between probability theory and deep learning that i have come across.
Re: Why does deep and cheap learning work so well?
#16Learning a transformation from the full transformation semigroup is the most general case. Consider a mystery unary CPU operation M on a 64 bit register. How deep of a circuit do you need to calculate any such transformation M? According to Kolmogorov just writing out a random transformation function takes (2^64)*(64) bits to make the lookup table. log of that to get a result out efficiently. Results here were already proved in information theory. If you use a depth less than log of the lookup table you are screwed unless your function is extremely non-random.
Re: Why does deep and cheap learning work so well?
#17Earlier quoted context omitted.
I think the GP poster meant to contrast this paper not with other papers, but with the junk explanations for "how deep learning works" that science journalism comes up with.
trust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves
Bollocks see my comment. We know worst case bounds for Kolmogorov random transformations of a finite set, the most complex learning task there is.
Re: Why does deep and cheap learning work so well?
#18One of my dreams is to do something similar by simplifying neural nets via (bi)simulation (something along the lines of quotient automata).
Re: Why does deep and cheap learning work so well?
#19Earlier quoted context omitted.
trust me some of the junk explanations for how deep learning works originate from researchers. we really just don't have a convincing picture about why these networks work or don't work in a theoretical way rather than empirical/heuristic handwaves
Bollocks see my comment. We know worst case bounds for Kolmogorov random transformations of a finite set, the most complex learning task there is.
huh? not even wrong......
Re: Why does deep and cheap learning work so well?
#20Same reason general relativity works. If you try to model results without a fundamental principle of the system's content's operation then you are going to have some limitations.
>"Same reason general relativity works."
Dark matter/energy?