Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation functions. It was picking wacky things like sine waves. You'd like to learn something about the model by the way it configures itself, but right now we can only understand simple model features.
Why does deep and cheap learning work so well?
21–30 of 53 posts
Re: Why does deep and cheap learning work so well?
#22It does not. It's about unsupervised learning and unlabeled data. It's the next frontier. Automated feature engineering for triangulated datasets is the real target.
Re: Why does deep and cheap learning work so well?
#23Same reason general relativity works. If you try to model results without a fundamental principle of the system's content's operation then you are going to have some limitations.
>"Same reason general relativity works." Dark matter/energy?
Well, technically general relativity has no real problems with either of those (there shouldn't be, given that general relativity is the main justification for both). It's the quantum mechanics behind them that's not really well understood.
Re: Why does deep and cheap learning work so well?
#24Love to see more papers like this. I remember a recent paper that showed decent model performance when the model was allowed to pick its own activation functions. It was picking wacky things like sine waves. You'd like to learn something about the model by the way it configures itself, but right now we can only understand simple model features.
That's so weird, I can't help but feel like that would only ever result in it coming up with something that initially got it a better result and the rest of the network adjusted around it and is able to carry the weight of that weirdness.
What IS interesting is that it shows just how resilient the networks that come out of this really are that they can support such weirdo activations.
But there are other elements of introspection that should absolutely be handled by neural nets, things like choosing how deep a given network needs to be should be accomplished by other neural nets specifically looking for trends while training.
Re: Why does deep and cheap learning work so well?
#25Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics: https://www.youtube.com/watch?v=5MdSE-N0bxs The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.
In that case, the fact that our minds run on similar substrates is probably non-coincidental.
Re: Why does deep and cheap learning work so well?
#26I'm not really sure about the "well" part, not even gonna to mention efficiency.
Re: Why does deep and cheap learning work so well?
#27One of my dreams is to do something similar by simplifying neural nets via (bi)simulation (something along the lines of quotient automata).
I think you will like this paper then: https://arxiv.org/abs/1711.09576
Re: Why does deep and cheap learning work so well?
#28I am not a Physicist (IANAP), so maybe I’m misrecalling some definitions, but isn’t the restriction to Hamiltonians a bit, well, restrictive? This limits the results to path independent potentials, which is basically nothing in the non-spherical cow world. Are the authors working from a different definition of Hamiltonian?
Does it matter if your Hamiltonian is smooth or if you are working from a discrete theory?
Re: Why does deep and cheap learning work so well?
#29> we show that n variables cannot be multiplied using fewer than 2^n neurons in a single hidden layer.
I don't know, but it just feels like this should have been known in CS earlier than 2016. Circuit complexity has been studied for a long time.
Re: Why does deep and cheap learning work so well?
#30> we show that n variables cannot be multiplied using fewer than 2^n neurons in a single hidden layer. I don't know, but it just feels like this should have been known in CS earlier than 2016. Circuit complexity has been studied for a long time.
Depends if you want an exact multiplication or not
Especially if you have neurons with non-linear response
remember ln(a*b) = ln(a)+ln(b)