It took me 10 years to understand entropy • Hacker News Board

Re: It took me 10 years to understand entropy

#161

Post by kgwgk » Thu, Apr 28, 2022, 1:41 PM UTC

IX-103 wrote: post #152

kgwgk wrote: post #123

Earlier quoted context omitted.

Sure, at least in principle. And if you knew what every molecule was doing the entropy would vanish.

I've thought that for a while, but I'm not a physicist. Do you know any prominent physicists that hold that view? It seems to contradict at least the popular narrative about entropy as a property of a system.

> Do you know any prominent physicists that hold that view?

The view that the entropy of a microstate (i.e. a perfectly defined physical state) is zero?

All of them, hopefully.

Re: It took me 10 years to understand entropy

#162

Post by sixo » Thu, Apr 28, 2022, 1:41 PM UTC

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> I don't know what Entropy is supposed to mean on the level of individual states/configurations. I don't understand what kind of macroscopic "averaging" function we may use to group up those states

I find it helpful to think of entropy as a property of not of the system, or any individual state (micro- or macro-), but as a property of the "compression" process that summarizes microstates with a coarser-grained macro-description.

Given a choice of compression, classical physics says a system will tend to spent most of their time in the most likely compressed state. Different choices of compressions can lead to different macro descriptions, with different "entropies" and different dynamics among their macroscopic variables.

In this light it's not meaningful to think of the entropy of individual states. You could think about the "identity" compression, but you would end up with a description that was exactly as complicated as the full micro-state time-evolution dynamics; you wouldn't end up with any smaller set of variables that could describe the equilibrium of the whole system (really this would not admit an "equilibrium" at all)

Re: It took me 10 years to understand entropy

#163

Post by oh_my_goodness » Thu, Apr 28, 2022, 1:42 PM UTC

kgwgk wrote: post #154

oh_my_goodness wrote: post #151

Earlier quoted context omitted.

It's different because I can define a macro-state without any information about which macro-state any system is actually in. As I think you're also saying, the only information I need is information about how I've defined my own macro-states. If we just define the macro-states, we're good to go. We don't need to talk about 'knowledge'. We can talk about 'knowledge', it's fine, but that lets in unnecessary woo.

I'm not sure I see what's the point of that distinction. The entropy of a macrostate is a measure of the indetermination about the microstate conditional on the macrostate. If you don't want to call that 'knowledge' the substance of the matter doesn't change. A macrostate is not an intrinsict property of a physical system. It's related to our description of the system. In general, the same microstate of the system of…

The point is pedagogical. Entropy takes a lot of time for people to understand clearly. That is the discussion from the OP.

Adding "knowlege" to the definition (or to an initial explanation) of entropy makes that learning process even more difficult. And it's unnecessary. It's better than the older talk about "disorder" but it's distracting. We can bypass 'knowledge' and come back later, with no penalty and plenty of time savings.

Apart from that single pedagogical point, we seem to be saying the same things back and forth to each other in different words. I'm not sure why.

Re: It took me 10 years to understand entropy

#164

Post by im3w1l » Thu, Apr 28, 2022, 1:45 PM UTC

mellavora wrote: post #122

The typical measure of entropy (Shannon or Gibbs, and let's spare details for later and after you've read up on the theory of large deviations) is - sum (p log(p)) which is not that different than the formula for the mean sum (p 1/n) the critical difference is the normalization constant is based on the probability of the state rather than assuming a uniform probability over all states. So, in effect, the entropy is a…

It can be related to compression. If some phrase has a probability p_i of occuring, then the optimal length for the code is -log(p_i). The entropy sum(-p_i log_pi) = mean(-log(p_i)) is how long code you will use on average.

Re: It took me 10 years to understand entropy

#165

Post by AnthonyAguirre » Thu, Apr 28, 2022, 1:47 PM UTC

Actually a quite nice article. After also spending years as a professional physicist not understanding entropy, I finally decided that I was not necessarily the problem, and spent the last 5 years or so trying to understand it better by rewording the foundations with my research group. (I'm one of the papers the author cites is part of a series from our group developing "observational entropy" in order to do so.)

A lot of what makes this topic confusing is just that there are the two basic definitions — Gibbs (\sum p_i log l_i) and "Boltzmann" (log \Omega) — entropy, and they're really rather different. There's usually some confusing handwaving about how to relate them, but the fact is that in a closed system one of them (generally) rises and the other doesn't, and one of them depends on a coarse-graining into macrostates and the other doesn't.

The better way to relate them, I've come to believe, is to consider them both as limits of a more general entropy (the one we developed — first in fact written down in some form by von Neumann but for some reason not pursued much over the years.) There's a brief version here: https://link.springer.com/article/10.1007/s10701-021-00498-x.

This entropy has Gibbs and Bolztmann entropy as limits, is good in and out of equilibrium, is defined in quantum theory and with a very nice classical-quantum correspondence, and has been shown to reproduce thermodynamic entropy in both our papers and the elegant one by Strasberg and Winter: https://journals.aps.org/prxquantum/abstract/10.1103/PRXQuan...

After all this work I finally feel that entropy makes sense to me, which it never quite did before — so I hope this is helpful to others.

p.s. If you're not convinced a new definition of entropy is called for, ask a set of working physicists what it would mean to say "the entropy of the universe is increasing." Since von Neumann entropy is conserved in a closed system (which the universe is if anything is), and there really is no definition of a quantum Boltzmann entropy (until observational entropy), the answers you'll get will be either a mush or a properly furrowed brows.

Re: It took me 10 years to understand entropy

#166

Post by stazz1 » Thu, Apr 28, 2022, 1:49 PM UTC

AnthonyAguirre wrote: post #165

Actually a quite nice article. After also spending years as a professional physicist not understanding entropy, I finally decided that I was not necessarily the problem, and spent the last 5 years or so trying to understand it better by rewording the foundations with my research group. (I'm one of the papers the author cites is part of a series from our group developing "observational entropy" in order to do so.) A l…

The universe is not a closed system

Re: It took me 10 years to understand entropy

#167

Post by kgwgk » Thu, Apr 28, 2022, 1:51 PM UTC

oh_my_goodness wrote: post #163

kgwgk wrote: post #154

Earlier quoted context omitted.

I'm not sure I see what's the point of that distinction. The entropy of a macrostate is a measure of the indetermination about the microstate conditional on the macrostate. If you don't want to call that 'knowledge' the substance of the matter doesn't change. A macrostate is not an intrinsict property of a physical system. It's related to our description of the system. In general, the same microstate of the system of…

The point is pedagogical. Entropy takes a lot of time for people to understand clearly. That is the discussion from the OP. Adding "knowlege" to the definition (or to an initial explanation) of entropy makes that learning process even more difficult. And it's unnecessary. It's better than the older talk about "disorder" but it's distracting. We can bypass 'knowledge' and come back later, with no penalty and plenty of…

I think that the "microstate counting" approach - if that's what you are defending - doesn't allow to understand entropy clearly because only works for the microcanonical description. It doesn't make sense to count the microstates for a volume of gas at some pressure and temperature. (Which is the standard thermodynamics problem.)

The concept of how much can we tell about the microstate given only the pressure and temperature seems quite natural and a better starting point. Boltzmann's entropy is a nice illustration but there is no reason to avoid the general concept.

Re: It took me 10 years to understand entropy

#168

Post by oh_my_goodness » Thu, Apr 28, 2022, 1:53 PM UTC

jbay808 wrote: post #160

oh_my_goodness wrote: post #142

Earlier quoted context omitted.

Susskind says that entropy is determined by selecting a macro-state. He doesn't claim that the entropy of a macro-state depends on whether we know which macro-state the real system is really in. If we happen to know, then, sure. For example we could pick a weird-ass observable state, and when we saw it we would know the entropy of the system was low. But the entropy of each macro-state just depends on how many micro-…

The concept of entropy wasn't invented so that we could calculate entropies of macrostates, it was so that we could calculate entropies of real systems and understand their behaviour. Macrostates are an accounting tool that helps us do this. You seem to be treating the calculation of macrostate entropy as an end-goal in itself, but also allowing yourself to somehow freely choose any macrostate you want. When it comes…

We seem to be stuck in a loop of explaining basic first-year statistical mechanics back and forth to each other repeatedly. I'm not sure why.

I'm making a pedagogical point. The OP addresses how difficult entropy is to understand. I'm responding to that. We don't need to talk about "knowledge" when you define entropy, or in an initial explanation of entropy. We could, but we could decide not to.

The log(x) example is a good one. First-time students who are learning about logarithms don't need to be told that a logarithm depends on 'knowledge' or on 'information.' It's ok to just tell them how logarithm is defined.

Sure, there is information. I'm saying it's confusing and unnecessary to introduce more big ideas like information, when the topic is "entropy is difficult to understand" or "logarithms are difficult to understand."

Re: It took me 10 years to understand entropy

#169

Post by oh_my_goodness » Thu, Apr 28, 2022, 2:00 PM UTC

oh_my_goodness wrote: post #163

kgwgk wrote: post #154

Earlier quoted context omitted.

I'm not sure I see what's the point of that distinction. The entropy of a macrostate is a measure of the indetermination about the microstate conditional on the macrostate. If you don't want to call that 'knowledge' the substance of the matter doesn't change. A macrostate is not an intrinsict property of a physical system. It's related to our description of the system. In general, the same microstate of the system of…

The point is pedagogical. Entropy takes a lot of time for people to understand clearly. That is the discussion from the OP. Adding "knowlege" to the definition (or to an initial explanation) of entropy makes that learning process even more difficult. And it's unnecessary. It's better than the older talk about "disorder" but it's distracting. We can bypass 'knowledge' and come back later, with no penalty and plenty of…

Of course you can count micro states of a gas within dE or delta-E of some total energy. The density-of-states approach is exactly that.

I thought we were discussing statistical mechanics.

Re: It took me 10 years to understand entropy

#170

Post by jbay808 » Thu, Apr 28, 2022, 2:00 PM UTC

im3w1l wrote: post #149

andi999 wrote: post #81

Earlier quoted context omitted.

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

I've been trying to reconcile these perspectives, and I think it really is both. And they are both physically relevant. Consider the subjective entropy perspective. If you know the exact microstate of a system, then you can in theory play the part of Maxwell's demon. You could have a little gate that you open only for fast particles, and using your knowledge of the microstate, you can predict exactly when they will a…

> If you take this very same system and put it in thermal contact with another system, then an objective entropy perspective is the relevant one. Those systems will equilibrize and your subjective knowledge is irrelevant to that process.

The subjective view handles this scenario just fine, though, and makes more accurate predictions than the objective view.

For example, there are systems where some aspects of the original microstate survive thermal contact with another system. We use such systems to store data! I bet your hard drive is in thermal contact with its environment right now! It's very hard to reconcile this with an objective take on entropy.

And there are some systems that will rapidly be scrambled. The subjective perspective has no problem admitting that your knowledge of a system can become inaccurate and useless. Even without thermal contact, you'd need to perform a tremendous amount of (perhaps reversible) computation in order to make a functioning Maxwell's demon with your initial microstate conditions, because the microstate will evolve in time in a complicated way. The subjective view is still totally consistent with entropy of a system increasing over time!