Where is Noether's principle in machine learning?

That's also a neat result! I'd just like to highlight that the conservation laws proved in that paper are functions of the parameters that hold over the course of gradient descent, whereas my post is talking about functions of the activations that are conserved from one layer to the next within an optimized network.

By the way, maybe I'm being too much of a math snob, but I'd argue Kunin's result is only superficially similar to Noether's theorem. (In the paper they call it a "striking similarity"!) Geometrically, what they're saying is that, if a loss function is invariant under a non-zero vector field, then the trajectory of gradient descent will be tangent to the codimension-1 distribution of vectors perpendicular to the vector field. If that distribution is integrable (in the sense of the Frobenius theorem), then any of its integrals is conserved under gradient descent. That's a very different geometric picture from Noether's theorem. For example, Noether's theorem gives a direct mapping from invariances to conserved quantities, whereas they need a special integrability condition to hold. But yes, it is a nice result, certainly worth keeping in mind when thinking about your gradient flows. :)

By the way, you might be interested in [1], which also studies gradient descent from the point of view of mechanics and seems to really use Noether-like results.

[1] Tanaka, Hidenori, and Daniel Kunin. “Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks.” In Advances in Neural Information Processing Systems, 34:25646–60. Curran Associates, Inc., 2021. https://papers.nips.cc/paper/2021/hash/d76d8deea9c19cc9aaf22....

It has been shown that a finite difference implementation of wave propagation can be expressed as a deep neural network (e.g., [1]). These networks can have thousands of layers and yet I don't think they suffer from the exploding/vanishing gradient problem, which I imagine is because in the physical system they model there are conservation laws such as conservation of energy.

[1] https://arxiv.org/abs/1801.07232

so I think this is a great connection that deserves more thought. as well as an absolutely gorgeous write-up.

The main problem I see with it is that most of the time you don't want the optimum for your objective function, as that frequently results in overfitting. this leads to things like early stopping being typical.

>at a high level--that life is a "reversal of the second law of thermodynamics";

Life temporarily displaces entropy, locally.

Life wins battles, chaos wins the war.

>Indeed, when considering machine learning, I think it's quite interesting to consider how the organizing of information/knowledge done during training in some real way mirrors the energy-creating information interred in the mind of Maxwell's demon.

This is our human bias favoring the common myth of ever-expanding complexity is an "inevitable" result of the passage of time; refer to Stephen Jay Gould's "Full House: The Spread of Excellence from Plato to Darwin"[0] for the only palatable refute modern evolutionists can offer.

>When taking into account the possible transitive benefits of knowledge organized via machine learning, and its attendant oracle through application, it's easy to see a world where this results in a net entropy loss, the creation of a previously non-existent energy gradient.

Because it is. Randomness combined with a sieve, like a generator and a discriminator, like the primordial protein soup and our own existence as a selector, like chaos and order themselves, MAY - but DOES NOT have to - lead to temporary, localized areas of complexity, that we call 'life'.

This "energy gradient" you speak of is literally gravity pulling baryonic matter foward thru space time. All work requires a temperature gradient - Hawking's musings on the second law of thermodynamics and your own intuition can reason why.

>In my mind this has interesting implications for Fermi's paradox as it seems to imply the inevitibility of the organization of information. Taken further into my own personal dogma, I think it's inevitable that we create--what we would consider--a sentient being as I believe this is the cycle of our own origin in the larger evolutionary timeline.

Over cosmological time spans, it is a near-mathematical certainty, that we are to either reach the universe's Omega point[1] on "our" own accord, perish to our own, by our own creation, or by our own son's, hands.

[0]: https://www.amazon.com/Full-House-Spread-Excellence-Darwin/d...

[1]: https://www.youtube.com/watch?v=eOxHRFN4rs0

It's been a long time since I have cracked a physics book, but your mention of interesting "fundamental physical quantities" triggered the recollection of there being a conservation of information result in quantum mechanics where you can come up with an action whose equations of motion are Schrödinger's equation and the conserved quantity is a probability current. So I wonder to what extent (if any) it might make sense to try to approach these things in terms of the really fundamental quantity of information itself?