Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

181–190 of 295 posts

Re: It took me 10 years to understand entropy

#181

Post by deltaonefour »

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

Another thing that contributes to the confusion which you have noted but not fully realized is that there are two different concepts that use the same equation and the same word: "entropy".

Information entropy and statistical entropy are two different things.

Re: It took me 10 years to understand entropy

#182

Post by liminvorous »

IX-103 wrote: post #152
kgwgk wrote: post #123

Earlier quoted context omitted.

Sure, at least in principle. And if you knew what every molecule was doing the entropy would vanish.

I've thought that for a while, but I'm not a physicist. Do you know any prominent physicists that hold that view? It seems to contradict at least the popular narrative about entropy as a property of a system.

I only have an undergraduate degree in physics, but I think the point you may be missing is that information requires some physical medium to store it.

So entropy is very roughly speaking the property of the system that determines how big a hard drive you need to store a description of that system.

Re: It took me 10 years to understand entropy

#183

Post by jbay808 »

coldtea wrote: post #180
jbay808 wrote: post #173

Earlier quoted context omitted.

But there's only one state that is 10010001111110101000.

Sure, but that's irrelevant. There are billions that are similar, and only one that's all 0.

"similar" is in the eye of the observer! This fact should be especially clear in the context of bit strings. If 10010001111110101000 is my login password, don't be surprised if other permutations fail to grant you access, even if you have the correct number of 1's and 0's.

Re: It took me 10 years to understand entropy

#184

Post by raxxorraxor »

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

In computer science a high entropy means a high informational value. So any information that is not the expected value has a high content of information and therefore a high entropy.

In thermodynamics the case is a bit to the opposite as a high entropy means a low state of energy and therefore less internal processes within a system or none at all.

Re: It took me 10 years to understand entropy

#185

Post by coldtea »

jbay808 wrote: post #183
coldtea wrote: post #180

Earlier quoted context omitted.

Sure, but that's irrelevant. There are billions that are similar, and only one that's all 0.

"similar" is in the eye of the observer! This fact should be especially clear in the context of bit strings. If 10010001111110101000 is my login password, don't be surprised if other permutations fail to grant you access, even if you have the correct number of 1's and 0's.

>"similar" is in the eye of the observer!

Nope, also similar in algorithmic complexity theory (e.g. counting compressibility).

Re: It took me 10 years to understand entropy

#186

Post by deltaonefour »

jbay808 wrote: post #73
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

> But if you look deeper than that averaging it stops making sense to me. It's a completely different world. I think you're less confused than you think you are! As I posted elsewhere, it helps to think of entropy as a quantity that actually depends on how much you know about the system in question. Typically when you calculate the entropy of a system at temperature X, that means all you know is that you stuck a ther…

No this is completely and utterly wrong. Entropy is not a function of knowledge.

Two people with varying and different levels of knowledge of a system does not mean the system has two different entropy values. Even if I knew the exact position of all atoms in a cup of water, the temperature of that water does not change due to that knowledge.

Entropy does rely on what your picked configuration of macro states and microstates. Temperature is an arbitrary choice of macrostate.

Re: It took me 10 years to understand entropy

#187

Post by jbay808 »

oh_my_goodness wrote: post #168
jbay808 wrote: post #160

Earlier quoted context omitted.

The concept of entropy wasn't invented so that we could calculate entropies of macrostates, it was so that we could calculate entropies of real systems and understand their behaviour. Macrostates are an accounting tool that helps us do this. You seem to be treating the calculation of macrostate entropy as an end-goal in itself, but also allowing yourself to somehow freely choose any macrostate you want. When it comes…

We seem to be stuck in a loop of explaining basic first-year statistical mechanics back and forth to each other repeatedly. I'm not sure why. I'm making a pedagogical point. The OP addresses how difficult entropy is to understand. I'm responding to that. We don't need to talk about "knowledge" when you define entropy, or in an initial explanation of entropy. We could, but we could decide not to. The log(x) example is…

Alright. I agree we seem to be stuck in a weird loop where we agree about all the observable facts but somehow are on different wavelengths in spite of that.

And I totally agree, entropy is a property of a macrostate. The information step comes inseparably in when you go from a system description to a macrostate. And you might just shuffle the confusion from not knowing what entropy is to not knowing what a macrostate is.

If you think it's clearer to teach students by explaining that the entropy of a macrostate is an objective property of that macrostate, that's fine. Just don't leave them believing that the entropy of a brick is an objective property of that brick.

Re: It took me 10 years to understand entropy

#189

Post by foobarqux »

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

For information theoretic entropy (I don’t know anything about thermodynamics):

The first thing you describe, that if you draw many symbols from your source distribution at random then you see a distribution of symbols that is equal to the probability distribution of the source, is called the asymptotic equipartition principle.

I think your confusion comes from two things. First, conflating bits and symbols and second assuming symbols are equiprobable.

Take English text where our symbols could be letters of the alphabet. These are not equiprobable, if you select a letter from a book at random you get a different distribution than 1/26. If you took as your symbols the individual bits that would encode those characters in ascii you would get something closer to equal probable symbols. Another choice for symbols would be the words in the book.

Re: It took me 10 years to understand entropy

#190

Post by deltaonefour »

jbay808 wrote: post #146
andi999 wrote: post #81

Earlier quoted context omitted.

Entropy (differences) are an objective quantity which can be measured, there is no subjectivity about it. It is not which parameters you know it is about which parameters you hold fixed.

No, it is subjective. We just only have such blunt instruments for practically measuring states, relative to the gargantuan amount of entropy in most real systems, that the subjective nature of entropy is easy to miss. But in a world where the frontiers of thermodynamics have moved from steam engines to lasers, computers, DNA, and black holes, the difference is increasingly obvious and important. With steam engines,…

Very few people know this but.

Information entropy and statistical mechanical entropy are two different things.

They share the same equation and the same name but they are two unrelated concepts. You have conflated the two. The person you are responding to is referring to statistical entropy.

Basically in this entire thread nobody, including you, is fully grasping the situation.

Post reply on HN