It took me 10 years to understand entropy • Hacker News Board

Re: It took me 10 years to understand entropy

#141

Post by oh_my_goodness » Thu, Apr 28, 2022, 12:35 PM UTC

vishnugupta wrote: post #82

jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

This is how I have come to understand entropy. The words disorder and order are a proxy for information content. > If you know more about the system, it has less entropy. One question though. When you say "it" does it include you as well as the system or just the system? To me "it" includes both because by it is "you" who's state has changed by acquiring more information. It could be in the form of neuronal rearrange…

>This is how I have come to understand entropy. The words disorder and order are a proxy for information content

Does information content mean this? ... "How many bits of random-number generator would I need to make the number of micro-states in the macro-state?"

Re: It took me 10 years to understand entropy

#142

Post by oh_my_goodness » Thu, Apr 28, 2022, 12:40 PM UTC

toxik wrote: post #114

andi999 wrote: post #81

Earlier quoted context omitted.

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

Leonard Susskind disagrees with you. See his lectures on statistical mechanics, he is very clear that entropy is a matter of knowledge about the system. It has to be.

Susskind says that entropy is determined by selecting a macro-state. He doesn't claim that the entropy of a macro-state depends on whether we know which macro-state the real system is really in.

If we happen to know, then, sure. For example we could pick a weird-ass observable state, and when we saw it we would know the entropy of the system was low. But the entropy of each macro-state just depends on how many micro-states we define it to contain. It doesn't depend on our knowledge of the system state.

Re: It took me 10 years to understand entropy

#143

Post by jbay808 » Thu, Apr 28, 2022, 12:43 PM UTC

oh_my_goodness wrote: post #139

jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

I think that works ok. But I think it's an unnecessarily tricky explanation. Entropy per macro-state decreases as we look at finer-grained macro-states. It feels simpler to associate the entropy of each macro-state with that macro-state, rather than assuming we know which macro-state the system is in, and then attributing the lower entropy to our knowledge of the macro-state. I think it can probably be expressed eith…

Can you elaborate on the difference between a "fine-grained" macrostate and a macrostate that is not fine-grained?

I think you will find it hard to separate the concept of a macrostate from the state of knowledge (or ignorance) of an individual subjective observer.

Re: It took me 10 years to understand entropy

#144

Post by oh_my_goodness » Thu, Apr 28, 2022, 12:48 PM UTC

jbay808 wrote: post #143

oh_my_goodness wrote: post #139

Earlier quoted context omitted.

I think that works ok. But I think it's an unnecessarily tricky explanation. Entropy per macro-state decreases as we look at finer-grained macro-states. It feels simpler to associate the entropy of each macro-state with that macro-state, rather than assuming we know which macro-state the system is in, and then attributing the lower entropy to our knowledge of the macro-state. I think it can probably be expressed eith…

Can you elaborate on the difference between a "fine-grained" macrostate and a macrostate that is not fine-grained? I think you will find it hard to separate the concept of a macrostate from the state of knowledge (or ignorance) of an individual subjective observer.

Sure. A fine-grained macro-state contains fewer micro-states. A coarse-grained macro-state contains more micro-states.

Say I flip 8 coins and I don't look at the results. A fine-grained macro state is TTTT TTTT. A coarser-grained macro state is TTTT xxxx. The one has 4 bits more entropy than the other. It works the same way in statistical mechanics. Call them spins.

We're just talking about some ensemble of micro states, and then we divide the ensemble up into macro-states. To do statistical mechanics at all, I think I have to define some macro-states according to which micro-states they contain. That doesn't mean I necessarily have any information about which macro-state the system is actually in.

Re: It took me 10 years to understand entropy

#145

Post by kgwgk » Thu, Apr 28, 2022, 1:00 PM UTC

oh_my_goodness wrote: post #144

jbay808 wrote: post #143

Earlier quoted context omitted.

Can you elaborate on the difference between a "fine-grained" macrostate and a macrostate that is not fine-grained? I think you will find it hard to separate the concept of a macrostate from the state of knowledge (or ignorance) of an individual subjective observer.

Sure. A fine-grained macro-state contains fewer micro-states. A coarse-grained macro-state contains more micro-states. Say I flip 8 coins and I don't look at the results. A fine-grained macro state is TTTT TTTT. A coarser-grained macro state is TTTT xxxx. The one has 4 bits more entropy than the other. It works the same way in statistical mechanics. Call them spins. We're just talking about some ensemble of micro sta…

How is a macrostate TTTTxxxx different from having the information about the TTTT part and not about the xxxx part?

Talking bout the entropy of the macrostate TTTTxxxx makes sense only conditional on the TTTT information.

Re: It took me 10 years to understand entropy

#146

Post by jbay808 » Thu, Apr 28, 2022, 1:05 PM UTC

andi999 wrote: post #81

jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

No, it is subjective. We just only have such blunt instruments for practically measuring states, relative to the gargantuan amount of entropy in most real systems, that the subjective nature of entropy is easy to miss. But in a world where the frontiers of thermodynamics have moved from steam engines to lasers, computers, DNA, and black holes, the difference is increasingly obvious and important.

With steam engines, we got away with treating a volume of gas as having not only a few parameters that we knew and cared about, like mass, temperature and pressure, but we could further deceive ourselves into thinking that those were the only parameters that existed to describe the system. The only parameters that were knowable. But Boltzmann knew better.

Look at Boltzmann's formula, S = kB log W.

For any single particular system you describe to me, W will be 1, and so S will be 0. So it's only if you describe an ensemble of systems -- that is, if you describe a system vaguely, such that I am left to imagine the details -- that we have nonzero entropy. If you ask me to calculate the entropy of that "system", that macrostate, that ensemble, then sure, I'll end up with nonzero entropy. But if I ask you to keep transmitting more data about the scenario, then with each further description, you'll be narrowing the state space and thereby decreasing the entropy.

Look, since the entropy of a macrostate is nonzero, but the entropy of any single microstate which is consistent with that macrostate is zero, it's clear that entropy is not an intrinsic property of any real system. It's a property of how many other possible non-existent systems could be swapped out for the one in front of you, without you noticing the change.

If I swap out the air in your room for an equal volume of air at equal temperature and pressure, you probably won't notice.

If I swap out the hard drive in your laptop for an equal volume of hard drive at equal temperature and pressure, you probably will!

Re: It took me 10 years to understand entropy

#147

Post by oh_my_goodness » Thu, Apr 28, 2022, 1:06 PM UTC

contravariant wrote: post #113

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

Yeah the author is conflating low entropy with a low number of microstates, which is consistent with the thermodynamic assumption that maximal entropy means a uniform distribution of microstates, but is confusing. The purest mathematical justification for why low entropy means a low number of microstates probably comes from the fact that (classical) physical systems are a dynamical systems that preserve the measure i…

>The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective.

That seems to be correct.

>Yeah the author is conflating low entropy with a low number of microstates

Since entropy is found by counting micro-states (for example your third paragraph), that should be ok. What am I missing?

Re: It took me 10 years to understand entropy

#148

Post by gota » Thu, Apr 28, 2022, 1:08 PM UTC

jbay808 wrote: post #73

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Oh my god, this explanation is gold. Thank you. I'm going to save it and refer to it in the future.

Re: It took me 10 years to understand entropy

#149

Post by im3w1l » Thu, Apr 28, 2022, 1:08 PM UTC

andi999 wrote: post #81

jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

I've been trying to reconcile these perspectives, and I think it really is both. And they are both physically relevant.

Consider the subjective entropy perspective. If you know the exact microstate of a system, then you can in theory play the part of Maxwell's demon. You could have a little gate that you open only for fast particles, and using your knowledge of the microstate, you can predict exactly when they will arrive.

But consider the objective perspective. If you take this very same system and put it in thermal contact with another system, then an objective entropy perspective is the relevant one. Those systems will equilibrize and your subjective knowledge is irrelevant to that process.

I haven't fully wrapped my head around it yet, but I do think that acknowledging both is a step in the right direction at least.

Re: It took me 10 years to understand entropy

#150

Post by eevilspock » Thu, Apr 28, 2022, 1:10 PM UTC

XorNot wrote: post #37

raven105x wrote: post #15

Entropy: "to describe energy loss in irreversible processes". We have no clue about what is or is not reversible. Complex systems exhibit self-organizing behavior for no reason (that we understand), and we continue to identify more conditions under which this occurs. How does a Nobel Prize get handed out for identifying/quantifying "self-organization" http://pespmc1.vub.ac.be/COMPNATS.html without bringing everything…

> Self-organization does not consume energy any more than entropic decay emits it. This statement is incredibly wrong - this is exactly what both these processes do. We calculate chemistry reaction kinetics by including entropy terms, and optimize reactions by manipulating the entropy on one side of the equation (a classic is getting a liquid phase to precipitate out as you produce it). I mean the reason coal can be…