Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where is Noether's principle in machine learning?

cgad.ski

Post reply on HN
Print view View original

51–60 of 81 posts

Re: Where is Noether's principle in machine learning?

#51

Post by empath-nirvana »

cgadski wrote: post #24
shiandow wrote: post #16

Earlier quoted context omitted.

A convolutional neural network ought to have translational symmetry, which should lead to a generalized version of momentum. If I understood the article correctly the conserved quantity would be , where dx is the finite difference gradient of x. This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.

Exactly right! In fact, because that symmetry does not include an action on the parameters of the layer, your conserved quantity should hold whether or not the network is stationary for a loss. This means that it'll be stationary on every single data point. (In an image classification model, these values are just telling you whether or not the loss would be improved if the input image were translated.)

Everything in the paper is talking about global symmetries, is there also the possibility of gauge symmetries?

Re: Where is Noether's principle in machine learning?

#52

Post by esafak »

irchans wrote: post #26

I liked the article and I hope that I can understand it more with some study. I think the following sentence in the article is wrong "Applying Noether's theorem gives us three conserved quantities—one for each degree of freedom in our group of transformations—which turn out to be horizontal, vertical, and angular momentum.” I think the correct statement is "Applying Noether's theorem gives us three conserved quantiti…

I'll be walking tall the day I can leisurely read articles like this! I wish I had studied this stuff; now time is short.

Re: Where is Noether's principle in machine learning?

#55

Post by jungturk »

empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

> now wondering...if there's some conserved quantity in the neural network that is _directly analagous_ to conserved quantities in physics

Isn't the model attempting to conserve information during training? And isn't information a physical quantity?

Re: Where is Noether's principle in machine learning?

#57

Post by Cleonis »

Aardwolf wrote: post #30
nostrademons wrote: post #15

Earlier quoted context omitted.

In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the a…

Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy? I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentu…

In retrospect: the earliest recognition of a conserved quantity was Kepler's law of areas. Isaac Newton later showed that Kepler's law of areas is a specific instance of a property that obtains for any central force, not just the (inverse square) law of gravity.

About symmetry under change of orientation: for a given (spherically symmetric) source of gravitational interaction the amount of gravitational force is the same in any orientation.

For orbital motion the motion is in a plane, so for the case of orbital motion the relevant symmetry is cilindrical symmetry with respect to the plane of the orbit.

The very first derivation that is presented in Newton's Principia is a derivation that shows that for any central force we have: in equal intervals of time equal amounts of area are swept out.

(The swept out area is proportional to the angular momentum of the orbiting object. That is, the area law anticipated the principle of conservation of angular momentum)

A discussion of Newton's derivation, illustrated with diagrams, is available on my website: http://cleonis.nl/physics/phys256/angular_momentum.php

The thrust of the derivation is that if the force that the motion is subject to is a central force (cilindrical symmetry) then angular momentum is conserved.

So: In retrospect we see that Newton's demonstration of the area law is an instance of symmetry-and-conserved-quantity-relation being used. Symmetry of a force under change of orientation has as corresponding conserved quantity of the resulting (orbiting) motion: conservation of angular momentum.

About conservation laws:

The law of conservation of angular momentum and the law of conservation of momentum are about quantities that are associated with specific spatial characteristics, and the conserved quantity is conserved over time.

I'm actually not sure about the reason(s) for classification of conservation of energy. My own view: we have that kinetic energy is not associated with any form of keeping track of orientation; the velocity vector is squared, and that squaring operation discards directional information. More generally, Energy is not associated with any spatial characteristic. Arguably Energy conservation is categorized as associated with symmetry under time translation because of absence of association with any spatial characteristic.

Re: Where is Noether's principle in machine learning?

#58

Post by jerf »

jcgrillo wrote: post #50
nostrademons wrote: post #38

Earlier quoted context omitted.

Right, but I'm saying that it's all modeling choices, all the way down. Extend the model to include thermal energy and most of the time it holds again - but then it falls down if you also have static electricity that generates a visible spark (say, a wool sweater on a slide) or magnetic drag (say, regenerative braking on a car). Then you can include models for those too, but you're introducing new concepts with each,…

It's been a long time since I have cracked a physics book, but your mention of interesting "fundamental physical quantities" triggered the recollection of there being a conservation of information result in quantum mechanics where you can come up with an action whose equations of motion are Schrödinger's equation and the conserved quantity is a probability current. So I wonder to what extent (if any) it might make se…

Approaching physics from a pure information flow is definitely a current research topic. I suspect we see less popsci treatment of it because almost nobody understands information at all, then trying to apply it to physics that also almost nobody understands is probably at least three or four bridges too far for a popsci treatment, but it's a current and active topic.

Re: Where is Noether's principle in machine learning?

#59

Post by jerf »

pvg wrote: post #18
r34 wrote: post #9

As a complete amateur I was wondering if it could be possible to use that property of light ("to always choose the most optimal route") to solve the traveling salesman problem (and the whole class of those problems as a consequence). Maybe not with an algorithmic approach, but rather some smart implementation of the machine itself.

Google up 'soap film steiner tree' for a fun, well-known variant of this.

Then follow it up with https://www.scottaaronson.com/papers/npcomplete.pdf . While reality can "solve" these problems to some extent it turns out that people overestimate reality's ability to solve it optimally.
Post reply on HN