What Is Entropy? • Hacker News Board

Re: What Is Entropy?

#121

Post by canadianfella » Thu, Apr 17, 2025, 5:46 PM UTC

overu589 wrote: post #115

quietbritishjim wrote: post #97

I like the axiomatic definition of entropy. Here's the introduction from Pattern Recognition and Machine Learning by C. Bishop (2006): > The amount of information can be viewed as the ‘degree of surprise’ on learning the value of x. If we are told that a highly improbable event has just occurred, we will have received more information than if we were told that some very likely event has just occurred, and if we knew…

How can that be axiomatic? I offer a coherent, concise dissenting view. Information is the removal of uncertainty. If it does not remove uncertainty it is not information. Uncertainty is state unresolved (potential resolves to state through constructive and destructive interference.) Entropy is the existential phenomenon of potential distributing over the infinite manifold of negative potential. “Uncertainty.” Emerge…

[dead]

Re: What Is Entropy?

#122

Post by FilosofumRex » Thu, Apr 17, 2025, 7:06 PM UTC

kgwgk wrote: post #120

FilosofumRex wrote: post #119

Earlier quoted context omitted.

More precisely, Von Neumann was extending Shannon's information theoretic entropy to quantum channels, which he restated as S(p)=Tr(p ln(p)) - Again showing that information theoretic entropy reveals nothing more about a system than its probability distribution density matrix p.

It’s quite remarkable that in his 1927 paper “The thermodynamics of quantum-mechanical ensembles” von Neumann was extending the mathematical theory of communication that Shannon - who was 11 at the time - would only publish decades later.

Dude, read your own reference... There is no mention of information or communication theory anywhere in his 1927 paper or 1932 book. Young Von Nuemann was doing real physics extending and updating Gibb's entropy.

OTOH, old Von Neumann was wealthy, hobnobbing with politicians and glitterati musing about life, biology, econ and anything else that would amuse his social circles. "Entropy", as he's alleged to have told Shannon, was his ace in the pocket to win arguments.

Formal similarity with Shannon's entropy is superfluous and conveys no new information about any system, quantum or otherwise. But it does make for lot's PhD dissertations, for exactly the same reason Von Nuemann stated.

Re: What Is Entropy?

#123

Post by kgwgk » Thu, Apr 17, 2025, 8:17 PM UTC

FilosofumRex wrote: post #122

kgwgk wrote: post #120

Earlier quoted context omitted.

It’s quite remarkable that in his 1927 paper “The thermodynamics of quantum-mechanical ensembles” von Neumann was extending the mathematical theory of communication that Shannon - who was 11 at the time - would only publish decades later.

Dude, read your own reference... There is no mention of information or communication theory anywhere in his 1927 paper or 1932 book. Young Von Nuemann was doing real physics extending and updating Gibb's entropy. OTOH, old Von Neumann was wealthy, hobnobbing with politicians and glitterati musing about life, biology, econ and anything else that would amuse his social circles. "Entropy", as he's alleged to have told S…

> There is no mention of information or communication theory anywhere in his 1927 paper or 1932 book. Young Von Nuemann was doing real physics extending and updating Gibb's entropy.

We agree then! John von Neumann’s work on entropy was about physics, not about communication theory. S(p)=Tr(p ln(p)) is physics. If you still claim that he “was extending Shannon's information theoretic entropy to quantum channels” at some point could you maybe give a reference?

> Formal similarity with Shannon's entropy is superfluous and conveys no new information about any system, quantum or otherwise

What I still don’t understand is your fixation with that.

“Entropy can't be a measure of uncertainty, because all the uncertainty is in the probability distribution p(x)” makes zero sense given that the entropy is a property of the probability distribution. (Any measure of “all the uncertainty” which is “in the probability distribution p(x)” will be a property of p(x). The entropy checks that box so why can’t it be a measure of uncertainty?)

It is a measure of the uncertainty in the probability distribution that describes a physical system in statistical mechanics. It is a measure of the lack of knowledge about the system. For a quantum system, von Neumann’s entropy becomes zero when the density matrix corresponds to a pure state and there is nothing left to know.