if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406 a refutation:(with very harsh/unprofessional words by tishby in counter-refutation) https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A…
I really like this kind of cross-disciplinary research and knowledge transfer. It should happen more often.
It’s interesting that so many equations governing different laws in different fields actually share quite a few properties, and on deeper analysis, can be explained by a single mathematical property. It makes me wonder how many insights we are missing simply because they were discovered in another field, under a different name, for a different purpose.
Here's a related 2016 talk by Max Tegmark (second author) on connections between deep learning and physics: https://www.youtube.com/watch?v=5MdSE-N0bxs The gist of it is that physical data tends to have symmetries, and these symmetries make descriptions of the data very compressible into relatively small neural circuits. Random data does not have this property, and cannot be learned easily. Super fascinating.
In that case, the fact that our minds run on similar substrates is probably non-coincidental.
Indeed, the connections seem profound. It seems to be a general-purpose optimal algorithm for, well, optimisation. And that would explain why the universe, our brains and AIs all trend toward it.
It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary.
Same reason general relativity works. If you try to model results without a fundamental principle of the system's content's operation then you are going to have some limitations.
>"Same reason general relativity works." Dark matter/energy?
Well that's actually my point, you are the one who got it
>"Same reason general relativity works." Dark matter/energy?
Well, technically general relativity has no real problems with either of those (there shouldn't be, given that general relativity is the main justification for both). It's the quantum mechanics behind them that's not really well understood.
They're posited as a fudge factor -because- the quantum mechanics is not understood.
In that case, the fact that our minds run on similar substrates is probably non-coincidental.
Indeed, the connections seem profound. It seems to be a general-purpose optimal algorithm for, well, optimisation. And that would explain why the universe, our brains and AIs all trend toward it. It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary.
> It could also be just that intelligence tends to mirror the outside world, but that seems a bit arbitrary.
We are part of the universe, so why would it be arbitrary if our brains were structured in ways that match typical structures found in this universe?
I am not a Physicist (IANAP), so maybe I’m misrecalling some definitions, but isn’t the restriction to Hamiltonians a bit, well, restrictive? This limits the results to path independent potentials, which is basically nothing in the non-spherical cow world. Are the authors working from a different definition of Hamiltonian? Does it matter if your Hamiltonian is smooth or if you are working from a discrete theory?
In this context, the Hamiltonian is just the log of the conditional probability of the input data given some parameters. In fact, such functions are much more diverse than the functions defined by neural networks (which, though there are many variants, are basically all compositions of linear and nonlinear functions). The question being asked in the paper is which Hamiltonians can be robustly approximated by neural networks. The authors then argue that the class of Hamiltonians that appear in nature are simple enough that neural nets do a decent job.
>"Same reason general relativity works." Dark matter/energy?
Well, technically general relativity has no real problems with either of those (there shouldn't be, given that general relativity is the main justification for both). It's the quantum mechanics behind them that's not really well understood.
>"the quantum mechanics behind them that's not really well understood."
if you are interested in this type of analysis are you familiar with the tishby papers? you will at the very least find them enjoyable/mind opening... but some of his key claims are probably contradicted by empirical results. the original paper: https://arxiv.org/abs/1503.02406 a refutation:(with very harsh/unprofessional words by tishby in counter-refutation) https://openreview.net/forum?id=ry_WPG-A-¬eId=ry_WPG-A…
I found his talk on youtube, remarkable: https://www.youtube.com/watch?v=XL07WEc2TRI
He's a good speaker/teacher but I think there are serious problems with his work- mainly that a lot of his results collapse when tested on more general architectures.