Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where is Noether's principle in machine learning?

cgad.ski

Post reply on HN
Print view View original

21–30 of 81 posts

Re: Where is Noether's principle in machine learning?

#21

Post by sdenton4 »

nostrademons wrote: post #15
empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the a…

A nice way to formulate (most) data augmentations is: a family of functions A = {a} such that our optimized neural network f obeys f(x) ~= f(a(x)).

So in this case, we're explicitly defining the set of desired invariances.

Re: Where is Noether's principle in machine learning?

#22

Post by sdenton4 »

PaulHoule wrote: post #13
dustingetz wrote: post #8

Earlier quoted context omitted.

Here's my ELI5 attempt of the time/energy relation: imagine a spring at rest (not moving) strike the spring, it's now oscillating the system now contains energy like a battery what is energy? it's stored work potential the battery is storing the energy, which can then be taken out at some future time the spring is transporting the energy through time in fact how do we measure time? with clocks. What's a clock? It's a…

Symmetry exists abstractly, apart from time. There are many machine learning problems which should have symmetries: a picture of a cow rotated 135 degrees is still a picture of a cow, the meaning of spoken words shouldn't change with the audio level, etc. If they were doing machine learning on tracks from the LHC the system ought to take account of relativistic momentum and energy. Can a model learn a symmetry? Or sh…

Equivariant machine learning is a thing that people have tried... Tends to be expensive and slow, though, and imposes invariances that our model (a universal function approximator, recall) should just learn anyway: If you don't have enough pictures of upside down cows, just train a normal model with augmentations.

Re: Where is Noether's principle in machine learning?

#23

Post by scarmig »

Scene_Cast2 wrote: post #17

People have mentioned the discrete - continuous tradeoff. One way to bridge that gap would be to use https://arxiv.org/abs/1806.07366 - they draw an equivalence between vanilla (FC layer) neural nets of constant width with differential equations, and then use a differential equation solver to "train" a "neural net" (from what I remember - it's been years since that paper...). Another approach might be to take an info…

Another angle to look at would be the S4 models, which admit both a continuous time and recurrent discrete representation.

Re: Where is Noether's principle in machine learning?

#24

Post by cgadski »

shiandow wrote: post #16
empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

A convolutional neural network ought to have translational symmetry, which should lead to a generalized version of momentum. If I understood the article correctly the conserved quantity would be , where dx is the finite difference gradient of x. This gives a vector with dimensions equal to however many directions you can translate a layer in and which is conserved over all (convolutional) layers.

Exactly right! In fact, because that symmetry does not include an action on the parameters of the layer, your conserved quantity should hold whether or not the network is stationary for a loss. This means that it'll be stationary on every single data point. (In an image classification model, these values are just telling you whether or not the loss would be improved if the input image were translated.)

Re: Where is Noether's principle in machine learning?

#25

Post by pmayrgundter »

I wonder if an energy and work metric could be derived for gradient descent. This might be useful for a more rigorous approach to hyperparameter development, and maybe for characterizing the data being learned. We say that some datasets are harder to learn, or measure difficulty by the overall compute needed to hit a quality benchmark. Something more essential would be a step forward.

Like in ANN backprop, the gradient descent algorithm can use a momentum to overcome getting stuck in local minima. This was heuristically physical when I learned it.. perhaps it's been developed since. Maybe only allowing a "real" energy to the momentum would then align it with an ability to do work calculation. Might also help with ensemble/monte carlo methods, to maintain an energy account across the ensemble.

Re: Where is Noether's principle in machine learning?

#26

Post by irchans »

I liked the article and I hope that I can understand it more with some study.

I think the following sentence in the article is wrong "Applying Noether's theorem gives us three conserved quantities—one for each degree of freedom in our group of transformations—which turn out to be horizontal, vertical, and angular momentum.”

I think the correct statement is "Applying Noether's theorem gives us three conserved quantities—one for each degree of freedom in our group of transformations—which turn out to be translation, rotation, and time shifting.”

I think translation leads to conservation of momentum, rotation leads to conservation of angular momentum, and time shifting leads to conservation of energy (potential+kinetic). It's been a few decades since I saw the proof, so I might be wrong.

Re: Where is Noether's principle in machine learning?

#27

Post by nrub »

platz wrote: post #4

how do you direct what the network learns if it all comes from supervised learning training sets? How do you insert rules that aren't learned into what weights are learned?

There are promising methods developing for Physic's informed neural networks. Mathematical models can be integrated into the architecture of neural networks such that the parameters of the designed mathematical models can be learned. Examples include learning the frequency of a swinging pendulum from video, amongst more advanced ideas.

https://en.wikipedia.org/wiki/Physics-informed_neural_networ... https://www.youtube.com/watch?v=JoFW2uSd3Uo

Re: Where is Noether's principle in machine learning?

#28

Post by [deleted] »

empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

[deleted]

Re: Where is Noether's principle in machine learning?

#30

Post by Aardwolf »

nostrademons wrote: post #15
empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the a…

Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy?

I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentum one and in space you have translation and rotation, so for time energy is the only one that's left over, but I'm not looking for a trick to remember it, I'm looking for the fundamental reason, as well as how to tell what will be paired with some invariance when looking at some other new invariance

Post reply on HN