Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

251–260 of 295 posts

Re: It took me 10 years to understand entropy

#251

Post by [deleted] »

deltaonefour wrote: post #181
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

Another thing that contributes to the confusion which you have noted but not fully realized is that there are two different concepts that use the same equation and the same word: "entropy". Information entropy and statistical entropy are two different things.

[deleted]

Re: It took me 10 years to understand entropy

#252

Post by kgwgk »

deltaonefour wrote: post #248
kgwgk wrote: post #227

Earlier quoted context omitted.

"accessible" means something only given a set of constraints. Like the temperature, if you keep the temperature of the water fixed. And the number of molecules if instead of a cup you have a close container to prevent it from evaporating. Then what you have is water at some temperature that you control. And you could have the water at a different temperature with exactly the same microstate. Or imagine gas at some fi…

>it wouldn't make sense to talk about its pressure - If I have a pressure gauge that reads the same thing regardless of my knowledge how is pressure meaningless? The tool that reads pressure gives me an accurate pressure number regardless of what I know or don't know. This number is correct. Your argument is basically saying that the pressure gauge becomes wrong once you have more knowledge of the system. No it doesn…

> The pressure gauge is still giving you a number defined as "pressure."

As long as you define “pressure” as “the reading of the manometer” and not as “the variable that together with temperature specifies the state of the gas and measures the quantity of energy required to compress it further”.

Thermodynamics is based on state variables giving a complete description of the system. Statistical mechanics is based on looking at the ensemble of microscopic descriptions possible given what is known about the system and their probabilities.

If all you know is a handful of thermodynamic variables that ensemble is huge. If you know already the microscopic description of the physical system your ensemble has one single possible configuration in it.

As in jbay808’s xkcd example, if you have a random number generator and you know the sequence of numbers that will be generated, do you have a random number generator? The random number generator is still giving you a number defined as “random”, right?

I guess that it’s still random if you “forget” that you know it in advance and that the macrostate is still meaningful as a complete description of the physical system if you “forget” that you have a perfect knowledge of its state.

Edit: the GPS receiver in my phone is giving me some coordinates defined as “position” that happen to be in the middle of the road. However, I know precisely where I am. Don’t you think that the meaning of that “position” is somehow affected by this additional information?

Re: It took me 10 years to understand entropy

#253

Post by lambdatronics »

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

>I don't know what Entropy is supposed to mean on the level of individual states/configurations.

The entropy is a property of a probability distribution, not of a state. Entropy is defined as H = -sum(p_i log(p_i)). A 'state' implicitly defines a probability distribution: uniform probability over all the microstates compatible with the state description.[0] In the case of a microstate, the entropy of the probability distribution over microstates consistent with that state is zero - there's only one compatible state, so p_i = 0 for all other states, and log(p_i) = 0 for the compatible state. In the case of a macrostate, the entropy of the probability distribution over microstates consistent with the macrostate works out to -sum((1/N) log(1/N)) = log(N), where N is the number of consistent microstates. That's the Boltzmann entropy.

Sometimes people will write about the entropy of a 'state' in such a way that it sounds like they're talking about the entropy of a microstate -- but what they're probably talking about is "the entropy of the macrostate that this microstate belongs to." It's sloppy to talk like that, because "the" corresponding macrostate isn't unique. There are many sets of macrostates that could contain a microstate, depending on what properties of the microstates one considers 'macro.'

(Ex: 10100101 is a member of both "symmetric bit strings of length 8" and "bit strings of length 8 that average to 1/2". The entropy of "symmetric bit strings of length 8" is 4 bits, whereas the entropy of "bit strings of length 8 that average to 1/2" is ~6.1 bits. And of course, the entropy of "the bit string of length 8 that is exactly 10100101" is zero.)

Re: It took me 10 years to understand entropy

#254

Post by lambdatronics »

d--b wrote: post #91
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

Not a physicist either, and I don't claim to understand entropy that well either but maybe it would help to consider that entropy may not be a universal variable of systems in the universe. I think you should rather consider it as a mathematical construct that applies to some systems where the microscopic quantities are well defined, and where the 'averaging' that we can observe is also well defined. So if you look a…

>It seems to say that "given a system we know everything about, there is no way to go to a system that has some unknown things to us".

That's backwards: information is the negative of entropy. The 2nd law says that entropy never decreases, so information never increases (it can only be preserved or lost).

Re: It took me 10 years to understand entropy

#255

Post by lambdatronics »

ogogmad wrote: post #59

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviatio…

>how physicists get around the discrete and finite restriction

By turning a sum into an integral. The probability 'density' is p(x), and the 'density of states' is n(x), so then entropy is then integral of p(x)log(p(x)/n(x)) over dx.

Re: It took me 10 years to understand entropy

#256

Post by lambdatronics »

ogogmad wrote: post #67
omegalulw wrote: post #65

Earlier quoted context omitted.

> I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). True, but for continuous distributions you can use the KL divergence against a uniform dist…

One of the properties of entropy H(X) of a random variable X is that if f is a bijective function then H(f(X)) = H(X). For relative entropy (or "KL divergence" as some people call it), we have that H(X||Y) = H(f(X)||f(Y)). But if you fix Y to have a continuous uniform distribution, then you lose this critical property because f(Y) may no longer have a continuous uniform distribution.

Yeah, you also have to transform the "reference" function, and then the entropy stays the same. I prefer to think of it as the "density of states" -- it's necessary to make the argument of the logarithm dimensionless, after all.

Re: It took me 10 years to understand entropy

#257

Post by omegalulw »

im3w1l wrote: post #245
omegalulw wrote: post #235

Earlier quoted context omitted.

+1. The commenter above also wanted cared about bijective mappings, and squaring a random variable in [-1, 1] is not bijective. Squaring a random variable defined over positive real numbers would lead to a bijective mapping and the distribution would still remain uniform. Actually, I find it hard to come up with a bijective mapping that leads to a non uniform distribution that's useful for anything practical.

Ok so first to have a uniform distribution we have to have a bounded set. Maybe you can do something clever with limits but lets not overcomplicate things. Lets say we have 0 Uniformity of v would mean that p(0 Uniformity of E would mean that p(0 But by construction p(0 <= v < 1) = p(0 <= E < 1). So it's not possible for both to be uniform.

It's not necessary to have p(0 But this does bring up a good point. H(X||Y) = H(f(X)||f(Y)) for any bijective f if the distributions are discrete. When they are continuous this is not true, even with a bijective f. For example f = x^2 doesn't work even though it yields a binary distribution. Interestingly however, affine transformations work.

Re: It took me 10 years to understand entropy

#258

Post by lambdatronics »

shannifin wrote: post #14

> "Entropy is not Disorder One of the most popular belief about entropy is that it represents disorder." This is what confused me the most about entropy in high school, the "order / disorder" lingo. Isn't "order" a metaphysical concept, something a conscious entity thinks about a system? How would nature know the difference? It took me some years to understand that that lingo is indeed misleading. (Still definitely n…

> Isn't "order" a metaphysical concept

Yes. The most scientific way of talking about "order" in that sense is the Kolmogorov complexity, which is still extremely poorly-defined. The best way to put it is that "ordered" states are ones that have low Kolmogorov complexity.

Re: It took me 10 years to understand entropy

#260

Post by im3w1l »

omegalulw wrote: post #257
im3w1l wrote: post #245

Earlier quoted context omitted.

Ok so first to have a uniform distribution we have to have a bounded set. Maybe you can do something clever with limits but lets not overcomplicate things. Lets say we have 0 Uniformity of v would mean that p(0 Uniformity of E would mean that p(0 But by construction p(0 <= v < 1) = p(0 <= E < 1). So it's not possible for both to be uniform.

It's not necessary to have p(0 But this does bring up a good point. H(X||Y) = H(f(X)||f(Y)) for any bijective f if the distributions are discrete. When they are continuous this is not true, even with a bijective f. For example f = x^2 doesn't work even though it yields a binary distribution. Interestingly however, affine transformations work.

(For non-negative v) v^2 is less than one if and only if v is less than one. That's why the probabilities have to agree in our specific case.
Post reply on HN