Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

91–100 of 221 posts

Re: What Is Entropy?

#91

Post by IIAOPSW »

glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

To shorten this for you with my own (identical) understanding: "entropy is just the name for the bits you don't have".

Entropy + Information = Total bits in a complete description.

Re: What Is Entropy?

#92

Post by IIAOPSW »

rachofsunshine wrote: post #30
glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).

Yeah but distributions are just the accounting tools to keep track of your entropy. If you are missing one bit of information about a system, your understanding of the system is some distribution with one bit of entropy. Like the original comment said, the entropy is the number of bits needed to fill in the unknowns and bring the uncertainty down to zero. Your coin flips may be unknown in advance to you, and thus you model it as a 50/50 distribution, but in a deterministic universe the bits were present all along.

Re: What Is Entropy?

#93

Post by kmeisthax »

waveBidder wrote: post #65
davidmnoll wrote: post #48

Earlier quoted context omitted.

Right but in chemistry class the way it’s taught via Gibbs free energy etc. makes it seem as if it’s an intrinsic property.

that's actually the normal view, with saying both info and stat mech entropy are the same is an outlier, most popularized by Jaynes.

If information-theoretical and statistical mechanics entropies are NOT the same (or at least, deeply connected) then what stops us from having a little guy[0] sort all the particles in a gas to extract more energy from them?

[0] https://en.wikipedia.org/wiki/Maxwell%27s_demon

Re: What Is Entropy?

#94

Post by [deleted] »

glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

[deleted]

Re: What Is Entropy?

#95

Post by CamperBob2 »

glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

It's an objective quantity, but you have to be very precise in stating what the quantity describes.

Unbroken egg? Low entropy. There's only one way the egg can exist in an unbroken state, and that's it. You could represent the state of the egg with a single bit.

Broken egg? High entropy. There are an arbitrarily-large number of ways that the pieces of a broken egg could land.

A list of the locations and orientations of each piece of the broken egg, sorted by latitude, longitude, and compass bearing? Low entropy again; for any given instance of a broken egg, there's only one way that list can be written.

Zip up the list you made? High entropy again; the data in the .zip file is effectively random, and cannot be compressed significantly further. Until you unzip it again...

Likewise, if you had to transmit the (uncompressed) list over a bandwidth-limited channel. The person receiving the data can make no assumptions about its contents, so it might as well be random even though it has structure. Its entropy is effectively high again.

Re: What Is Entropy?

#97

Post by klysm »

prof-dr-ir wrote: post #20

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…

The definition by itself without intuition of application is of little use

Re: What Is Entropy?

#98

Post by penguin_booze »

BigParm wrote: post #13
Jun8 wrote: post #3

A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that…

Von Neumann was the king of kings

He's a certified Martian: https://en.wikipedia.org/wiki/The_Martians_(scientists).

Re: What Is Entropy?

#99

Post by [deleted] »

galaxyLogic wrote: post #73
ajkjk wrote: post #53

Earlier quoted context omitted.

Well it's part of math, which physics is already based on. Whereas metaphysics is, imo, "stuff that's made up and doesn't matter". Probably not the most standard take.

I'm wondering, isn't Information Theory as much part of physics as Thermodynamics is?

[deleted]

Re: What Is Entropy?

#100

Post by kgwgk »

galaxyLogic wrote: post #73
ajkjk wrote: post #53

Earlier quoted context omitted.

Well it's part of math, which physics is already based on. Whereas metaphysics is, imo, "stuff that's made up and doesn't matter". Probably not the most standard take.

I'm wondering, isn't Information Theory as much part of physics as Thermodynamics is?

Would you say that Geometry is as much a part of physics as Optics is?
Post reply on HN