Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where is Noether's principle in machine learning?

cgad.ski

Post reply on HN
Print view View original

71–80 of 81 posts

Re: Where is Noether's principle in machine learning?

#71

Post by calhoun137 »

Aardwolf wrote: post #30
nostrademons wrote: post #15

Earlier quoted context omitted.

In physics, the conserved quantity isn't always time. Invariance over time translation is specifically conservation of energy. Invariance over spatial translation is conservation of momentum, invariance over spatial rotation is conservation of conservation of angular momentum, invariance of electromagnetic field is conservation of current, and invariance of wave function phase is conservation of charge. I think the a…

Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy? I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentu…

The key point is that energy, momentum, and angular momentum are additive constants of the motion, and this additivity is a very important property that ultimately derives from the geometry of the space-time in which the motion takes place.

> Is there any way to deduce which invariance gives which conservation?

Yes. See Landau vol 1 chapter 2 [1].

> I'm looking for the fundamental reason, as well as how to tell what will be paired with some invariance when looking at some other new invariance

I'm not sure there is such a "fundamental reason", since energy, momentum, and angular momentum are by definition the names we give to the conserved quantities associated with time, translation, and rotation.

You are asking "how to tell what will be paired with some invariance" but this is not at all obvious in the case of conservation of charge, which is related to the fact that the results of measurements do not change when all the wavefunctions are shifted by a global phase factor (which in general can depend on position).

I am not aware of any way to guess or understand which invariance is tied to which conserved quantity other than just calculating it out, at least not in a way that is intuitive to me.

[1] https://ia803206.us.archive.org/4/items/landau-and-lifshitz-...

Re: Where is Noether's principle in machine learning?

#72

Post by Aardwolf »

calhoun137 wrote: post #71
Aardwolf wrote: post #30

Earlier quoted context omitted.

Is there any way to deduce which invariance gives which conservation? I mean for example: how can you tell that time invariance is the one paired with conservation of energy? Why is e.g. time invariance not paired with momentum, current, or anything else, but specifically energy? I know that I can remember momentum is paired with translation simply because there's both the angular momentum and the non-angular momentu…

The key point is that energy, momentum, and angular momentum are additive constants of the motion, and this additivity is a very important property that ultimately derives from the geometry of the space-time in which the motion takes place. > Is there any way to deduce which invariance gives which conservation? Yes. See Landau vol 1 chapter 2 [1]. > I'm looking for the fundamental reason, as well as how to tell what…

But momentum is also conserved over time, as far as I know 'conservation' of all of these things always means over time.

"In a closed system (one that does not exchange any matter with its surroundings and is not acted on by external forces) the total momentum remains constant."

That means it's conserved over time, right? So why is energy the one associated with time and not momentum?

Re: Where is Noether's principle in machine learning?

#73

Post by calhoun137 »

Aardwolf wrote: post #72
calhoun137 wrote: post #71

Earlier quoted context omitted.

The key point is that energy, momentum, and angular momentum are additive constants of the motion, and this additivity is a very important property that ultimately derives from the geometry of the space-time in which the motion takes place. > Is there any way to deduce which invariance gives which conservation? Yes. See Landau vol 1 chapter 2 [1]. > I'm looking for the fundamental reason, as well as how to tell what…

But momentum is also conserved over time, as far as I know 'conservation' of all of these things always means over time. "In a closed system (one that does not exchange any matter with its surroundings and is not acted on by external forces) the total momentum remains constant." That means it's conserved over time, right? So why is energy the one associated with time and not momentum?

my understandinf is that conservation of momentum does not mean momentum is conserved as time passes. it means if you have a (closed) system in a certain configuration (not in an external field) and compute the total momentum, the result is independent of the configuration of the system.

Re: Where is Noether's principle in machine learning?

#74

Post by danbmil99 »

I need to digest this but it is a seductive idea. My quick take: there may be a connection between back-propagation and reversibility, both computational and physical. For a system to be reversible implies conservation of information.

It also makes me think about the surprising success of highly quantized models (see for example recent paper on ternary networks, where the only valid numbers re 0, 1, and -1.)

Artificial Neural Networks were originally conceived as an approximation to an analog, continuous system, where floating-point numbers are stand-ins for reals. This is related to the ability to back-prop because real functions are generally differentiable. But if it turns out that we can closely approximate the same behavior with a small, discrete set of integers, it makes the whole edifice feel more like some sort of Cellular Automaton with reversible rules, rather than a set of functions over the reals.

Finally (sorry for the rabbit-holing) - how does this relate to our brains? Note that real neurons "fire" -- that is, they generate a discrete event when their internal configuration reaches a triggering state.

Lots to chew on...

Re: Where is Noether's principle in machine learning?

#75

Post by danbmil99 »

nurple wrote: post #39
empath-nirvana wrote: post #2

This is one of those links where just seeing the title sets you off, thinking about the implications. I'm going to have to spend more time digesting the article, but one thing that jumps out at me, and maybe it's answered in the article and I don't understand it, is the role of time. Generally in physics, you're talking about a quantity being conserved over time, and I'm not sure what plays the role of time when you'…

I think the most profound insight I've come across while studying this particular topic is the insight that information theory ended up being the answer to conserving the 2nd law with respect to Maxwell's demon thought experiment. Not to put too fine a point, but essentially the knowledge organized in the mind of the demon, about the particles in its system, was calculated to offset the creation of the energy gradien…

I like the connection to Fermi. It seems to me eventually there has to be a concrete answer to the following question: Given the laws of physics, and the "initial conditions" (ie the state of the Universe at the moment of the Big Bang), what is the statistical likelihood of advanced (ie technological) civilizations occurring over time, and what is the likelihood that they go extinct (or revert to less technologically savvy conditions)? ISTM there are intrinsic numbers for this calculation, though it is probably impossible for us to derive them from first principles.

Re: Where is Noether's principle in machine learning?

#76

Post by ogogmad »

jerf wrote: post #58
jcgrillo wrote: post #50

Earlier quoted context omitted.

It's been a long time since I have cracked a physics book, but your mention of interesting "fundamental physical quantities" triggered the recollection of there being a conservation of information result in quantum mechanics where you can come up with an action whose equations of motion are Schrödinger's equation and the conserved quantity is a probability current. So I wonder to what extent (if any) it might make se…

Approaching physics from a pure information flow is definitely a current research topic. I suspect we see less popsci treatment of it because almost nobody understands information at all, then trying to apply it to physics that also almost nobody understands is probably at least three or four bridges too far for a popsci treatment, but it's a current and active topic.

This might be insultingly simplistic, but I always thought the phrase "conservation of information" just meant that the time-evolution operator in quantum mechanics was unitary. Unitary mappings are always bijective functions - so it makes intuitive sense to say that all information is preserved. However, it does not follow that this information is useful to actually quantify, like energy or momentum. There is certainly a kind of applied mathematics called "information theory", but I doubt there's any relevance to the term "conservation of information" as it's used in fundamental physics.

The links below lend credibility to my interpretation.

https://en.wikipedia.org/wiki/Time_evolution#In_quantum_mech...

https://en.wikipedia.org/wiki/Bijection

https://en.wikipedia.org/wiki/Black_hole_information_paradox

Re: Where is Noether's principle in machine learning?

#77

Post by pyinstallwoes »

danbmil99 wrote: post #74

I need to digest this but it is a seductive idea. My quick take: there may be a connection between back-propagation and reversibility, both computational and physical. For a system to be reversible implies conservation of information. It also makes me think about the surprising success of highly quantized models (see for example recent paper on ternary networks, where the only valid numbers re 0, 1, and -1.) Artifici…

Kinda like the reversibility of chained xor’s and it’s ability to preserve information through cyclical permutations?

Re: Where is Noether's principle in machine learning?

#78

Post by danbmil99 »

pyinstallwoes wrote: post #77
danbmil99 wrote: post #74

I need to digest this but it is a seductive idea. My quick take: there may be a connection between back-propagation and reversibility, both computational and physical. For a system to be reversible implies conservation of information. It also makes me think about the surprising success of highly quantized models (see for example recent paper on ternary networks, where the only valid numbers re 0, 1, and -1.) Artifici…

Kinda like the reversibility of chained xor’s and it’s ability to preserve information through cyclical permutations?

Yes, that is roughly correct. You nay want to look up "reversible computation". It's a fundamental part of quantum computing, for one thing.

The key insight is that a (finite) discrete, reversible system will always eventually cycle back to its original state. This fact has very interesting follow-on implications for the concept of entropy and the Second Law. If it is guaranteed that a system will return to a prior state, how can it also be true that entropy (disorder) always increases?

Re: Where is Noether's principle in machine learning?

#79

Post by rnhmjoj »

Aardwolf wrote: post #72
calhoun137 wrote: post #71

Earlier quoted context omitted.

The key point is that energy, momentum, and angular momentum are additive constants of the motion, and this additivity is a very important property that ultimately derives from the geometry of the space-time in which the motion takes place. > Is there any way to deduce which invariance gives which conservation? Yes. See Landau vol 1 chapter 2 [1]. > I'm looking for the fundamental reason, as well as how to tell what…

But momentum is also conserved over time, as far as I know 'conservation' of all of these things always means over time. "In a closed system (one that does not exchange any matter with its surroundings and is not acted on by external forces) the total momentum remains constant." That means it's conserved over time, right? So why is energy the one associated with time and not momentum?

Conservation normally means things don't change over time just because in mechanics time is the go to external parameter to study the evolution of a system, but it's not the only one, nor the most convenient in some cases.

In Hamiltonian mechanics there is a 1:1 correspondence between any function of the phase space (coordinates and momenta) and one-parameter continous transformations (flows). If you give me a function f(q,p) I can construct some transformation φ_s(q,p) of the coordinates that conserves f, meaning d/ds f(φ_s(q, p)) = 0. (Keeping it very simple, the transformation consists in shifting the coordinates along the lines tangent to the gradient of f.)

If f(q,p) is the Hamiltonian H(q,p) itself, φ_s turns out to be the normal flow of time, meaning φ_s(q₀,p₀) = (q(s), p(s)), i.e. s is time and dH/dt = 0 says energy is conserved, but in general f(q,p) can be almost anything.

For example, take geometric optics (rays, refraction and such things): it's possible to write a Hamiltonian formulation of optics in which the equations of motion give the path taken by light rays (instead of particle trajectories). In this setting time is still a valid parameter but is most likely to be replaced by the optical path length or by the wave phase, because we are interested in steady conditions (say, laser turned on, beam has gone through some lenses and reached a screen). Conservation now means that quantities are constants along the ray, an example may be the frequency/color, which doesn't change even when changing between different media.

Re: Where is Noether's principle in machine learning?

#80

Post by rnhmjoj »

calhoun137 wrote: post #73
Aardwolf wrote: post #72

Earlier quoted context omitted.

But momentum is also conserved over time, as far as I know 'conservation' of all of these things always means over time. "In a closed system (one that does not exchange any matter with its surroundings and is not acted on by external forces) the total momentum remains constant." That means it's conserved over time, right? So why is energy the one associated with time and not momentum?

my understandinf is that conservation of momentum does not mean momentum is conserved as time passes. it means if you have a (closed) system in a certain configuration (not in an external field) and compute the total momentum, the result is independent of the configuration of the system.

It certainly means that momentum is conserved as time passes. The variation of the total momentum of a system is equal to the impulse, which is zero if there are no external fields.
Post reply on HN