Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

111–120 of 221 posts

Re: What Is Entropy?

#111

Post by enugu »

petsfed wrote: post #33
dekhn wrote: post #9

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged ( https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula ) although I found it easiest to think in terms of pairs of dice rolls.

And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties . Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.

Assuming each of the N microstates for a given macrostate are equally possible with probability p=1/N, the Shannon Entropy is -Σp.log(p) = -N.p.log(p)=-1.log(1/N)=log(N), which is the physics interpretation.

In the continuous version, you would get log(V) where V is the volume in phase space occupied by the microstates for a given macrostate.

Liouville's theorem that the volume is conserved in phase space implies that any macroscopic process can only move all the microstates from a macrostate A into a macrostate B only if the volume of B is bigger than the volume of A. This implies that the entropy of B should be bigger than the entropy of A which is the Second Law.

Re: What Is Entropy?

#113

Post by kergonath »

enugu wrote: post #110
kergonath wrote: post #108

Earlier quoted context omitted.

Entropy in Physics is usually either the Boltzmann or Gibbs entropy, both of whom were dead before Shannon was born.

That's not a problem, as the GP's post is trying to state a mathematical relation not a historical attribution. Often newer concepts shed light on older ones. As Baez's article says, Gibbs entropy is Shannon's entropy of an associated distribution(multiplied by the constant k).

It is a problem because all three come with a bagage. Almost none of the things discussed in this thread are invalid when discussing actual physical entropy even though the equations are superficially similar. And then there are lots of people being confidently wrong because they assume that it’s just one concept. It really is not.

Re: What Is Entropy?

#114

Post by bsmith »

bubblyworld wrote: post #104
canjobear wrote: post #78

Earlier quoted context omitted.

What's often lost in the discussions about whether entropy is subjective or objective is that, if you dig a little deeper, information theory gives you powerful tools for relating the objective and the subjective. Consider cross entropy of two distributions H[p, q] = -Σ p_i log q_i. For example maybe p is the real frequency distribution over outcomes from rolling some dice, and q is your belief distribution. You can…

Thanks, that's an interesting perspective. It also highlights one of the weak points in the concept, I think, which is that this is only a tool for updating beliefs to the extent that the underlying probability space ("ontology" in this analogy) can actually "model" the phenomenon correctly! It doesn't seem to shed much light on when or how you could update the underlying probability space itself (or when to change y…

Couldn't you just add a control (PID/Kalman filter/etc) to coverage on a stability of some local "most" truth?

Re: What Is Entropy?

#115

Post by ctafur »

ctafur wrote: post #112

The way I understand it is with an analogy to probability. To me, events are to microscopic states like random variable is to entropy.

My first contact with entropy was in chemistry and thermodynamics and I didn't get it. Actually I didn't get anything from engineering thermodynamics books such as Çengel and so.

You must go to statistical mechanics or information theory to understand entropy. Or trying these PRICELESS NOTES from Prof. Suo: https://docs.google.com/document/d/1UMwpoDRZLlawWlL2Dz6YEomy...

Re: What Is Entropy?

#118

Post by kgwgk »

petsfed wrote: post #33
dekhn wrote: post #9

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged ( https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula ) although I found it easiest to think in terms of pairs of dice rolls.

And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties . Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.

> not very compatible with Shannon's usage in any but the shallowest sense

The connection is not so shallow, there are entire books based on it.

“The concept of information, intimately connected with that of probability, gives indeed insight on questions of statistical mechanics such as the meaning of irreversibility. This concept was introduced in statistical physics by Brillouin (1956) and Jaynes (1957) soon after its discovery by Shannon in 1948 (Shannon and Weaver, 1949). An immense literature has since then been published, ranging from research articles to textbooks. The variety of topics that belong to this field of science makes it impossible to give here a bibliography, and special searches are necessary for deepening the understanding of one or another aspect. For tutorial introductions, somewhat more detailed than the present one, see R. Balian (1991-92; 2004).”

https://arxiv.org/pdf/cond-mat/0501322

Re: What Is Entropy?

#119

Post by enugu »

kergonath wrote: post #113
enugu wrote: post #110

Earlier quoted context omitted.

That's not a problem, as the GP's post is trying to state a mathematical relation not a historical attribution. Often newer concepts shed light on older ones. As Baez's article says, Gibbs entropy is Shannon's entropy of an associated distribution(multiplied by the constant k).

It is a problem because all three come with a bagage. Almost none of the things discussed in this thread are invalid when discussing actual physical entropy even though the equations are superficially similar. And then there are lots of people being confidently wrong because they assume that it’s just one concept. It really is not.

Don't see how the connection is superficial. Even the classical macroscopic definition of entropy as ΔS=∫TdQ can be derived from the information theory perspective as Baez shows in article(using entropy maximizing distributions and Lagrange multipliers). If you have a more specific critique, it would be good to discuss.

Re: What Is Entropy?

#120

Post by strogonoff »

bee_rider wrote: post #70
tonetegeatinst wrote: post #61

Earlier quoted context omitted.

Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.

He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about. A funny quote about him from a Edward “a guy with multiple equations named after him” Teller: > Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I somet…

Are there many von-Neumann-like multidisciplinaries nowadays? It feels like unless one is razor sharp fully into one field one is not to be treated seriously by those who made careers in it (and who have the last word on it).
Post reply on HN