Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

171–180 of 295 posts

Re: It took me 10 years to understand entropy

#171

Post by vishnugupta »

oh_my_goodness wrote: post #141
vishnugupta wrote: post #82

Earlier quoted context omitted.

This is how I have come to understand entropy. The words disorder and order are a proxy for information content. > If you know more about the system, it has less entropy. One question though. When you say "it" does it include you as well as the system or just the system? To me "it" includes both because by it is "you" who's state has changed by acquiring more information. It could be in the form of neuronal rearrange…

>This is how I have come to understand entropy. The words disorder and order are a proxy for information content Does information content mean this? ... "How many bits of random-number generator would I need to make the number of micro-states in the macro-state?"

That is my mental model, yes. More bits are needed to capture more detailed (or micro-states as you called it elsewhere in this thread, or finer-grained) information.

Let's say there's a stone, we want to know its details. If all we want to know is whether it weighs more than 100KG or not then one bit will do. 1 means > 100KG and 0 means This is just for the storage though; in order to gain the information we need to expend energy. More information requires more energy leading to more disorder as expending energy releases heat and thus 2nd order of thermodynamics as well as arrow of time. IMO our perception of time is purely based on memory which is information content of event stored in Neurons.

Quite a bit of hand-wavy. But this is a mental model I've developed over the years of thinking and reading (and listening to lectures) about entropy, information, arrow of time, and energy and how they are interconnected.

Re: It took me 10 years to understand entropy

#172

Post by im3w1l »

kgwgk wrote: post #102
ogogmad wrote: post #67

Earlier quoted context omitted.

One of the properties of entropy H(X) of a random variable X is that if f is a bijective function then H(f(X)) = H(X). For relative entropy (or "KL divergence" as some people call it), we have that H(X||Y) = H(f(X)||f(Y)). But if you fix Y to have a continuous uniform distribution, then you lose this critical property because f(Y) may no longer have a continuous uniform distribution.

Apparently this "critical property" is not so important to all the people who use relative entropy as a generalization to a continuous distribution defined on a space with an underlying measure. Why would they care about arbitrary transformations mapping points in the space to other points in the space?

What I think it means, is that if you take two different parametrizations of the same physical phenomenon, then you get two different entropy values.

E.g. if you have a bunch of particles with fixed mass. You could look at the distribution of speeds and get one entropy. Then the distribution of kinetic energy (basically speed squared). Uniform speed means non-uniform speed squared so the entropies would disagree.

This sounds like it could pose issues.

Re: It took me 10 years to understand entropy

#173

Post by jbay808 »

coldtea wrote: post #135
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. Well, there are only 2 states with all 1 or all 0. But there are 2^N states of mixed 1 and 0. Even if you treat the sets of bits as opaque items, and…

But there's only one state that is 10010001111110101000.

Re: It took me 10 years to understand entropy

#174

Post by kgwgk »

oh_my_goodness wrote: post #169
oh_my_goodness wrote: post #163

Earlier quoted context omitted.

The point is pedagogical. Entropy takes a lot of time for people to understand clearly. That is the discussion from the OP. Adding "knowlege" to the definition (or to an initial explanation) of entropy makes that learning process even more difficult. And it's unnecessary. It's better than the older talk about "disorder" but it's distracting. We can bypass 'knowledge' and come back later, with no penalty and plenty of…

Of course you can count micro states of a gas within dE or delta-E of some total energy. The density-of-states approach is exactly that. I thought we were discussing statistical mechanics.

> I thought we were discussing statistical mechanics.

This is from the message that you first replied to in this thread:

"Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a thermometer in it and measured X. You don't know anything more than the average temperature. It could be in any state consistent with that temperature."

Will you tell students to count the microstates consistent with the temperature?

Re: It took me 10 years to understand entropy

#176

Post by contravariant »

oh_my_goodness wrote: post #147
contravariant wrote: post #113

Earlier quoted context omitted.

Yeah the author is conflating low entropy with a low number of microstates, which is consistent with the thermodynamic assumption that maximal entropy means a uniform distribution of microstates, but is confusing. The purest mathematical justification for why low entropy means a low number of microstates probably comes from the fact that (classical) physical systems are a dynamical systems that preserve the measure i…

>The trick is that all of this is true no matter how you partition phase-space. Though that does mean that what is and isn't a high entropy state depends on your perspective. That seems to be correct. >Yeah the author is conflating low entropy with a low number of microstates Since entropy is found by counting micro-states (for example your third paragraph), that should be ok. What am I missing?

The equivalence is an important theorem (important enough to be engraved on Boltzmann's gravestone), and if you're switching back and forth in an explanation of what entropy is then you're skipping over some important details that answer what it means for something to be a 'low entropy state'.

Re: It took me 10 years to understand entropy

#177

Post by danbruc »

Banana699 wrote: post #138
danbruc wrote: post #126

Earlier quoted context omitted.

First, entropy is a macroscopic property, it makes no sense to talk about the entropy of a single particle. Second, entropy is not a fundamental property, it depends on what the observer cares about. Take the common example of a gas in the corner of a box, in that case we care about the density distribution in the box, a macroscopic property. To make this more concrete, one way to quantify the density distribution co…

>there are much more possibilities even though they are microscopically indistinguishable. You meant "macroscopically" ?

True, but to late to edit.

Re: It took me 10 years to understand entropy

#178

Post by alecst »

andi999 wrote: post #81
jbay808 wrote: post #73

Earlier quoted context omitted.

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

U, the internal energy is objective.

The free energy F = U - TS is the maximum amount of work you can extract from the system. This depends on how much you know about the system. S does indeed depend on what you know about the system.

See the Gibbs Paradox for more information.

Re: It took me 10 years to understand entropy

#179

Post by rytill »

brabel wrote: post #131
alimov wrote: post #76

Earlier quoted context omitted.

> The article describes it as a measure of hidden information in a system, which is a good description. But that's not a property of the system itself, it's a property of the observer, from whom the information is hidden. Was hoping to see someone point this bit out.. I wish references to entropy included this piece of information more frequently. When I was first trying to understand the concept I kept thinking of i…

Speaking of observers always rubs me off the wrong way... I don't want to touch on the observer problem, but just to mention something that should be obvious: there's ALWAYS hidden information in any system where time exists. Any "observer" can only know what the world looks like within its light cone. Because quantum mechanics shows that determinism is not possible, it's not possible for any "observer" to know the e…

Whatever is the driving factor behind the laws of physics seems to have “perfect information”. Not implying anything religious.

Re: It took me 10 years to understand entropy

#180

Post by coldtea »

jbay808 wrote: post #173
coldtea wrote: post #135

Earlier quoted context omitted.

> Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. Well, there are only 2 states with all 1 or all 0. But there are 2^N states of mixed 1 and 0. Even if you treat the sets of bits as opaque items, and…

But there's only one state that is 10010001111110101000.

Sure, but that's irrelevant.

There are billions that are similar, and only one that's all 0.

Post reply on HN