Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

61–70 of 221 posts

Re: What Is Entropy?

#61

Post by tonetegeatinst »

BigParm wrote: post #13
Jun8 wrote: post #3

A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that…

Von Neumann was the king of kings

Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.

Re: What Is Entropy?

#62

Post by [deleted] »

davidmnoll wrote: post #48
rachofsunshine wrote: post #30

Earlier quoted context omitted.

This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).

Right but in chemistry class the way it’s taught via Gibbs free energy etc. makes it seem as if it’s an intrinsic property.

[deleted]

Re: What Is Entropy?

#63

Post by yellowcake0 »

Information entropy is literally the strict lower bound on how efficiently information can be communicated (expected number of transmitted bits) if the probability distribution which generates this information is known, that's it. Even in contexts such as calculating the information entropy of a bit string, or the English language, you're just taking this data and constructing some empirical probability distribution from it using the relative frequencies of zeros and ones or letters or n-grams or whatever, and then calculating the entropy of that distribution.

I can't say I'm overly fond of Baez's definition, but far be it from me to question someone of his stature.

Re: What Is Entropy?

#64

Post by slashdave »

Lichtso wrote: post #57
dekhn wrote: post #9

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged ( https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula ) although I found it easiest to think in terms of pairs of dice rolls.

The "can be arranged" is the tricky part. E.g. you might know from context that some states are impossible (where the probability distribution is zero), even though they combinatorially exist. That changes the entropy to you. That is why information and entropy are different things. Entropy is what you know you do not know. That knowledge of the magnitude of the unknown is what is being quantified. Also, the point wh…

Exactly. If you want to reuse the term "entropy" in information theory, then fine. Just stop trying to make a physical analogy. It's not rigorous.

Re: What Is Entropy?

#65

Post by waveBidder »

davidmnoll wrote: post #48
rachofsunshine wrote: post #30

Earlier quoted context omitted.

This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).

Right but in chemistry class the way it’s taught via Gibbs free energy etc. makes it seem as if it’s an intrinsic property.

that's actually the normal view, with saying both info and stat mech entropy are the same is an outlier, most popularized by Jaynes.

Re: What Is Entropy?

#67

Post by canjobear »

davidmnoll wrote: post #48
rachofsunshine wrote: post #30

Earlier quoted context omitted.

This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).

Right but in chemistry class the way it’s taught via Gibbs free energy etc. makes it seem as if it’s an intrinsic property.

Entropy in physics is usually the Shannon entropy of the probability distribution over system microstates given known temperature and pressure. If the system is in equilibrium then this is objective.

Re: What Is Entropy?

#70

Post by bee_rider »

tonetegeatinst wrote: post #61
BigParm wrote: post #13

Earlier quoted context omitted.

Von Neumann was the king of kings

Its odd...as someone interested but not fully into the sciences I see his name pop up everywhere.

He was really brilliant, made contributions all over the place in the math/physics/tech field, and had a sort of wild and quirky personality that people love telling stories about.

A funny quote about him from a Edward “a guy with multiple equations named after him” Teller:

> Edward Teller observed "von Neumann would carry on a conversation with my 3-year-old son, and the two of them would talk as equals, and I sometimes wondered if he used the same principle when he talked to the rest of us."

Post reply on HN