Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

231–240 of 295 posts

Re: It took me 10 years to understand entropy

#231

Earlier quoted context omitted.

> are there any cases where it would it be impossible to empirically demonstrate that at least the proponent of the lower figure was wrong? Isn't it more interesting to examine a situation where it would be possible to empirically demonstrate that the proponent of the lower figure was wrong?

In that case we could objectively say that one value of S does not yield F for that system (given that F is defined as a maximum), but this would not resolve the general question of subjectivity.

If that doesn't, I'm not sure what would. Maybe it would help if I taboo the word "subjective". Are you familiar with Maxwell's demon? Let's set up a variation of that experiment.

I have a partitioned box full of air at room temperature and pressure in both partitions. There's a frictionless door that can be open and closed by an ultrafast servomechanism. The servo is connected to a computer which will read a very long bitstring from a magnetic hard-drive platter at a high frequency and open the door when the bit is '1' and close it when the bit is '0'.

Admittedly, this mechanism would be hard to construct in practice, but I hope it's clear enough as a thought experiment.

Now if you're familiar with Maxwell's demon, you'll agree that there are particular, albeit rare, joint configurations of gas microstate and hard drive bitstring, such that after the servo has finished its last motion, the gas will have been separated into hot and cold on either side of the partition. This temperature difference can be used to extract work.

For each possible bitstring on the drive, there are certain corresponding microstates of gas that will maximize the free energy extracted by this process.

And for each possible microstate of the gas, there are certain corresponding bitstrings that will maximize the free energy extracted by this process.

(For the vast majority of other combinations of hard drive bitstring and gas microstate, the operation will have no effect).

The claim "entropy is subjective" is basically just an acknowledgement that the energy extractable from the gas is dependent on both the state of the gas itself and also the data written to the hard drive. It means that two experimenters, tasked with writing the initial data on the hard drive to extract as much work as possible from the gas, will have different levels of success depending on whether they know the particular microstate of the gas (and can thus select the corresponding optimal bitstring) or if they don't know the microstate of the gas beyond "a box at room temperature and pressure", and have to guess a bitstring based on only that. And when the operation is successful, we can describe the data on the hard drive as "information about the gas microstate that was used to extract work".

This experiment, of course, is so impractical that it sounds ridiculous. But we can make a more controlled version of it on the small scale, with excited trapped atoms, and actually make it work.

Re: It took me 10 years to understand entropy

#232

Earlier quoted context omitted.

A thermometer doesn't measure temperature any better than a meterstick measures length. And we all know what Einstein had to say about the relativity of metersticks. To paraphrase from the paper I linked in another reply to you, a thermometer is just a heat bath equipped with a pointer which reads its average energy, whose scale is calibrated to give the temperature T, defined by 1/T = dS/d . You can read the thermom…

But entropy does not change with this knowledge.

You seem pretty convinced. Let me see if you're talking about the same pedantic distinction that oh_my_goodness was.

A: "an urn containing either a white ball or a black ball".

B: "I notice that the ball in the urn A is white".

I would say that initially the entropy of our ball-urn system is 1 bit, and that with observation B, we have reduced the entropy of our ball-urn system to 0 bits.

But if you are going to take the view that even knowing the ball in this particular urn is actually white doesn't change the fact that the entropy of > is 1 bit and not 0, and that that's the entropy that we're discussing, then I won't argue about it any further.

Except for the obligatory xkcd: https://xkcd.com/221/

Re: It took me 10 years to understand entropy

#233

Earlier quoted context omitted.

The article does say that some crystalline structures can have more entropy (information) than their fluid state. How could that be? Any ideas on what that fluid state might be? The information content in a crystal is really low.

Unfortunately the author doesn't explain it beyond sharing a reference to this paper[1] which is way beyond my competence. [1] https://www.nature.com/articles/nature08641

Beyond me too, but I'm going to assume that this is sort of an edge case where fluid crystalline structure can have less entropy than the static version (sort of sounds like the laminar flow state has less complex structure than its packed solid state). I doubt it contradicts your description above (which is similar to my understanding as well).

Re: It took me 10 years to understand entropy

#234
post #172
post #102

Earlier quoted context omitted.

Apparently this "critical property" is not so important to all the people who use relative entropy as a generalization to a continuous distribution defined on a space with an underlying measure. Why would they care about arbitrary transformations mapping points in the space to other points in the space?

What I think it means, is that if you take two different parametrizations of the same physical phenomenon, then you get two different entropy values. E.g. if you have a bunch of particles with fixed mass. You could look at the distribution of speeds and get one entropy. Then the distribution of kinetic energy (basically speed squared). Uniform speed means non-uniform speed squared so the entropies would disagree. Thi…

Physical entropy is defined from the probability distribution over states. Velocities or squared-velocities are not states, they are derived quantities. Points in a phase space would describe states. Physical states are discrete anyway when you consider quantum physics :-)

As for the entropy of probability distributions in general, I think relative entropy is invariant under reparametrizations because both the probability of interest and the reference probability transform in the same way [1]. But I don't remember what does it mean exactly. [And I am not sure if that makes ogogmad wrong, I may not have understood well his comment.]

([Edit: forget this aside. You probably were talking about speeds as positive magnitudes.] By the way using an example analogue to yours discrete entropy wouldn't be invariant either: if you have a distribution {-1,1} and square it it collapses to a zero-entropy singleton {1}.)

[1] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Pr...

Re: It took me 10 years to understand entropy

#235
post #234
post #172

Earlier quoted context omitted.

What I think it means, is that if you take two different parametrizations of the same physical phenomenon, then you get two different entropy values. E.g. if you have a bunch of particles with fixed mass. You could look at the distribution of speeds and get one entropy. Then the distribution of kinetic energy (basically speed squared). Uniform speed means non-uniform speed squared so the entropies would disagree. Thi…

Physical entropy is defined from the probability distribution over states. Velocities or squared-velocities are not states, they are derived quantities. Points in a phase space would describe states. Physical states are discrete anyway when you consider quantum physics :-) As for the entropy of probability distributions in general, I think relative entropy is invariant under reparametrizations because both the probab…

+1. The commenter above also wanted cared about bijective mappings, and squaring a random variable in [-1, 1] is not bijective. Squaring a random variable defined over positive real numbers would lead to a bijective mapping and the distribution would still remain uniform.

Actually, I find it hard to come up with a bijective mapping that leads to a non uniform distribution that's useful for anything practical.

Re: It took me 10 years to understand entropy

#236

The typical measure of entropy (Shannon or Gibbs, and let's spare details for later and after you've read up on the theory of large deviations) is - sum (p log(p)) which is not that different than the formula for the mean sum (p 1/n) the critical difference is the normalization constant is based on the probability of the state rather than assuming a uniform probability over all states. So, in effect, the entropy is a…

Perhaps I'm misunderstanding or missing something, but I'm afraid this seems completely wrongheaded to me. (My apologies for being so blunt, but right now your comment appears to be the most-upvoted, and I therefore think it needs some pushback.)

[EDITED to add: I was looking at an old version of the page; by the time I wrote this the parent was no longer the top comment. I'll leave the bluntness in, especially as at least one other person was even blunter.]

You refer to "the mean" and I think you mean the mean of the probabilities. Now, when you've got a probability distribution, by far the usual thing for "the mean" to mean is the sum of Pr(x) x -- the mean of the values. Taking the mean of the probabilities is a really strange thing to do.

One reason why it's a really strange thing to do is that this thing you call n is really kinda meaningless. There's no difference between these two probability distributions: (a) 1, 2, 3, or 4, with probabilities 0.1, 0.2, 0.3, 0.4 respectively; (b) 1, 2, 3, 4, or 5, with probabilities 0.1, 0.2, 0.3, 0.4, 0 respectively. But (a) has n=4 and (b) has n=5. Maybe you want n to be the number of nonzero probabilities? But now consider (a) along with the following probability distribution parameterized by a (small, positive) number h: 1, 2, 3, 4, or 4+h, with probabilities 0.1, 0.2, 0.3, 0.4-h, h. Every version of this distribution with h>0 has n=5, but when h is very small it's practically indistinguishable from (a) with n=4.

Further, since the sum of probabilities is always 1, what you write as sum (p 1/n) is just the same as the number 1/n. You can call it "the mean" if you want to, but I don't see what this adds over calling it what it is: the reciprocal of the number of possibilities.

There is something to what you say: the entropy is kinda related to the number of possibilities; if the probabilities are all equal, the entropy is log(#possibilities); if the probabilities are equal-ish then it's modestly smaller than that. But note e.g. that this relationship is exactly the inverse of what you say, in that "the mean" decreases with the number of possibilities, and the entropy increases with the number of possibilities.

The entropy is not "a measure of the mean". It kinda-sorta is related to "the number of possibilities", which is the reciprocal of "the mean". It is not at all the case, as your last paragraph suggests, that for most purposes we should be using "the mean" but we need to use the entropy when "the number of modes ... is not handled well by simpler metrics", whatever that means; for most purposes we should be using the entropy, and in the special case where all the probabilities are equal we can get away with just counting possibilities.

(In some important situations it turns out that what you have is some number of possibilities with roughly equal probabilities, and a whole lot more whose probabilities rapidly decrease to almost zero, and then you can get away with counting the number of reasonably-probable possibilities and taking its log. E.g., various situations in communications theory can fruitfully be thought of this way. But the entropy is still the more fundamental quantity, and "the mean" is still a needless obfuscation of "the (effectively) number of possibilities".)

Re: It took me 10 years to understand entropy

#237

Earlier quoted context omitted.

In that case we could objectively say that one value of S does not yield F for that system (given that F is defined as a maximum), but this would not resolve the general question of subjectivity.

If that doesn't, I'm not sure what would. Maybe it would help if I taboo the word "subjective". Are you familiar with Maxwell's demon? Let's set up a variation of that experiment. I have a partitioned box full of air at room temperature and pressure in both partitions. There's a frictionless door that can be open and closed by an ultrafast servomechanism. The servo is connected to a computer which will read a very lo…

> If that doesn't, I'm not sure what would.

It is not clear to me in which direction you think the question would be resolved, given a situation in which it would be possible to empirically demonstrate that the proponent of the lower figure (for S) was wrong. Maybe that is because I do not see the connection between your thought experiment and this issue: neither of the candidate values for S entail a particular distribution of states, let alone a particular sequence of future times when a molecule will approach the gate in a particular direction.

As I see it, your experiment is a difficult-to-perform way to demonstrate that, due to the inherent randomness of thermal processes, the entropy of a closed system may decrease when the conditions are right. This is explicitly covered in the article (see also "Monkeys typing Hamlet.")

Furthermore, in the case where the microstates of the system are measured in detail and the arrival times and velocities of the molecules at the gate are computed, one must add in the change in entropy resulting from those measurements and calculations. I am pretty sure this has been done, and is in accordance with the 2nd. law.

Your definition of 'subjective' in your penultimate paragraph is contrary to both common usage and what is being discussed in this thicket of threads, and appears to be closer to 'stochastic'. The outcome of the spin of a roulette wheel does not become subjective when different gamblers place different bets on it, or even when someone who has recorded statistics for its outcomes is able to place better-than-random bets.

Re: It took me 10 years to understand entropy

#238

The author mentions Boltzman brains and that a human body could theoretically spontaneously form out of particles given a long enough time span. Of course, nothing like this can ever happen. It’s the fallacy of thinking infinite time means infinite possibilities.

Why is it not bound to happen eventually?

For the same reason an infinite number of zeroes will never contain a one. Particles still have to follow the laws of how matter behaves, no matter the timescale. Nothing spontaneously forms like the author suggests, regardless of entropy. That’s not how matter works.

Re: It took me 10 years to understand entropy

#239
post #149

Earlier quoted context omitted.

I've been trying to reconcile these perspectives, and I think it really is both. And they are both physically relevant. Consider the subjective entropy perspective. If you know the exact microstate of a system, then you can in theory play the part of Maxwell's demon. You could have a little gate that you open only for fast particles, and using your knowledge of the microstate, you can predict exactly when they will a…

> If you take this very same system and put it in thermal contact with another system, then an objective entropy perspective is the relevant one. Those systems will equilibrize and your subjective knowledge is irrelevant to that process. The subjective view handles this scenario just fine, though, and makes more accurate predictions than the objective view. For example, there are systems where some aspects of the ori…

I wrote two replies to this that I both deleted. Then I had a good long thunk, and here's what I came up with.

The temperature of an object can be determined through 1/T = dS / dE. What is this S? How can it exist if you know the system perfectly? And here is where the great insight comes. The thermometer! You apply a thermometer to a system you perturb it! The system may have started in one particular microstate, but the very nature of thermal contact involves random influence. Those random tiny influences from the thermometer allow the object (harddrive in our case) to enter a bunch of microstates with certain probabilities. And that's what S measures.

So our subjective knowledge does actually not matter. (Classically speaking) the system is in a particular microstate we may know it or not, and it still manages to have a temperature. That is due to the states it could hypothetically enter (but haven't yet)!

If we think back to the harddrive and it's contents: Very gently touching a harddrive with a thermometer while not scramble its contents. So we may say that microstates corresponding to different files than the ones you put there are actually not accessible. And they don't contribute to the entropy we used for the temperature.

Re: It took me 10 years to understand entropy

#240

The author mentions Boltzman brains and that a human body could theoretically spontaneously form out of particles given a long enough time span. Of course, nothing like this can ever happen. It’s the fallacy of thinking infinite time means infinite possibilities.

It's not a fallacy, it's a paradox. One that indicates that our theories of quantum fluctuations in an infinite universe are incomplete. This podcast episode has a good discussion on the subject: https://universetoday.fireside.fm/745

The initial conversation is an entirely false premise, that an infinitely large universe would have "anything that can happen, would happen." An infinitely large universe could be empty, and fit the bill, and there would be nothing like "there are infinite copies of myself that are slightly different than now." It's bad philosophy, not science.
Post reply on HN