Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

It took me 10 years to understand entropy

cantorsparadise.com

Post reply on HN
Print view View original

261–270 of 295 posts

Re: It took me 10 years to understand entropy

#261

Post by esalman »

threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

High entropy = less predictable system. Low entropy = highly predictable system.

Re: It took me 10 years to understand entropy

#262

Post by chermi »

jbay808 wrote: post #160
oh_my_goodness wrote: post #142

Earlier quoted context omitted.

Susskind says that entropy is determined by selecting a macro-state. He doesn't claim that the entropy of a macro-state depends on whether we know which macro-state the real system is really in. If we happen to know, then, sure. For example we could pick a weird-ass observable state, and when we saw it we would know the entropy of the system was low. But the entropy of each macro-state just depends on how many micro-…

The concept of entropy wasn't invented so that we could calculate entropies of macrostates, it was so that we could calculate entropies of real systems and understand their behaviour. Macrostates are an accounting tool that helps us do this. You seem to be treating the calculation of macrostate entropy as an end-goal in itself, but also allowing yourself to somehow freely choose any macrostate you want. When it comes…

I think you might enjoy cosma shalizi's paper "What is a macrostate?" https://arxiv.org/abs/cond-mat/0303625

Re: It took me 10 years to understand entropy

#263

Post by chermi »

deltaonefour wrote: post #190
jbay808 wrote: post #146

Earlier quoted context omitted.

No, it is subjective. We just only have such blunt instruments for practically measuring states, relative to the gargantuan amount of entropy in most real systems, that the subjective nature of entropy is easy to miss. But in a world where the frontiers of thermodynamics have moved from steam engines to lasers, computers, DNA, and black holes, the difference is increasingly obvious and important. With steam engines,…

Very few people know this but. Information entropy and statistical mechanical entropy are two different things. They share the same equation and the same name but they are two unrelated concepts. You have conflated the two. The person you are responding to is referring to statistical entropy. Basically in this entire thread nobody, including you, is fully grasping the situation.

They are not at all unrelated. It is not easy to grasp, so I understand the confusion. https://en.m.wikipedia.org/wiki/Landauer%27s_principle

Fun rabbit hole would start with classic paper by jaynes

Many more recent examples relating bit erasure costs of computation. Some names to look up if interested include charlie Bennet,Dave wolpert, James crutchfield, Susanne still, for starters.

Edit -- a collection of ideas related to this problem and mixing in "complexity" can be found in SFI proceedings called "Complexity, entropy, and the physics of information"

Re: It took me 10 years to understand entropy

#264

Post by chermi »

mathgenius wrote: post #100
threatripper wrote: post #61

I don't understand entropy and this article did not change it. The issue I take is with the definition of "the most likely state". Think of a series of random bits that can be either 0 or 1 with equal probability. How likely is it that they are all 0 or all 1? Not very likely. There is exactly one configuration. How likely is it that they have a specific configuration of 0 and 1? Equally likely. All states are equall…

Have you looked at Huffman coding? I'd recommend the (free) book by David MacKay, it is secretly the "hackers guide to thermodynamics". http://www.inference.org.uk/mackay/itila/book.html

This book is phenomenal and I highly second this recommendation.

Re: It took me 10 years to understand entropy

#265

Post by chermi »

evouga wrote: post #25

One aspect of entropy that I always find counterintuitive is that unlike mass, charge, etc. it is not a physical quantity. In fact, from the point of view of an experimenter with perfect information about a physical system, the entropy of the system is exactly conserved over time (as made precise by Liouville's Theorem). The Second Law survives in this setting only in the most trivial sense that a constant function d…

I would ask why you don't have the same problem with energy?

Re: It took me 10 years to understand entropy

#266

Post by chermi »

ogogmad wrote: post #59

I thought entropy (in the Shannon sense) was a property of discrete and finite probability distributions. It's essentially a measure of how random a sample from such a probability distribution is. Notably, continuous probability distributions don't have meaningful entropy (or in some sense, their entropy is always infinite). It's worth considering the similarities and differences between entropy and standard deviatio…

You basically nailed it!

Re: It took me 10 years to understand entropy

#267

Post by chermi »

SnowHill9902 wrote: post #13

Statistical mechanics is one way of representing entropy but you don’t need it. The second law of thermodynamics can be expressed in other much more general terms. Also it requires that the system be isolated not “thermally isolated”. There’s other types of interactions such as gravitational and electromagnetic.

Can you explain what you mean by "not needing stat mech" and thermo entropy being "more general"?

Re: It took me 10 years to understand entropy

#268

Post by deltaonefour »

kgwgk wrote: post #252
deltaonefour wrote: post #248

Earlier quoted context omitted.

>it wouldn't make sense to talk about its pressure - If I have a pressure gauge that reads the same thing regardless of my knowledge how is pressure meaningless? The tool that reads pressure gives me an accurate pressure number regardless of what I know or don't know. This number is correct. Your argument is basically saying that the pressure gauge becomes wrong once you have more knowledge of the system. No it doesn…

> The pressure gauge is still giving you a number defined as "pressure." As long as you define “pressure” as “the reading of the manometer” and not as “the variable that together with temperature specifies the state of the gas and measures the quantity of energy required to compress it further”. Thermodynamics is based on state variables giving a complete description of the system. Statistical mechanics is based on l…

>Edit: the GPS receiver in my phone is giving me some coordinates defined as “position” that happen to be in the middle of the road. However, I know precisely where I am. Don’t you think that the meaning of that “position” is somehow affected by this additional information?

No it is not affected by it. The meaning of position is never changed. Your knowledge of your position can change, but your actual position exists regardless of your knowledge or inaccuracies of your tools.

>As in jbay808’s xkcd example, if you have a random number generator and you know the sequence of numbers that will be generated, do you have a random number generator? The random number generator is still giving you a number defined as “random”, right?

Random number generators are a rabbit hole. There's not even a proper mathematical definition for it. We're not sure what a random number is... we just have an intuition for it. Case in point, the xkcd article could not define it mathematically. This is the reason why the joke exists, because we're not even truly sure what it is or if random numbers are a thing. We have intuition for what a random number is but this is likely some kind of illusion similar to the many optical illusions produced by our visual cortex. If formalization of our intuitions are not possible then there is likelihood that the intuition is not even real.

>Statistical mechanics is based on looking at the ensemble of microscopic descriptions possible given what is known about the system and their probabilities.

ok take a look at this: https://math.stackexchange.com/questions/2916887/shannon-ent...

They're talking about deriving the entropy formula for fair dice. But they talk about it as if we don't have knowledge about physics, momentum and projectile motion. We have the power to simulate the dice in a computer simulation and know the EXACT outcome of the dice. The dice is a cube and easily modeled with mathematics. So then why does the above discussion even exist? What is the point of fantasizing about dice as if we have no knowledge of how to mechanically calculate the outcome? The point is they chose a specific set of macrostates that have uniform distribution across all the outcomes. It is a choice that is independent of knowledge.

Re: It took me 10 years to understand entropy

#269

Post by XorNot »

andrewflnr wrote: post #242
XorNot wrote: post #39

Earlier quoted context omitted.

"Compressibility" (in the software sense) would perhaps carry the most meaning. Entropy is ultimately all about the ability to extract information (i.e. work) out of a system.

"Compressibility" is really not that great an illustration for physical entropy. Information-theoretic entropy is not quite the same thing as physical entropy, but close enough to confuse you if you're not paying attention.

I'm going to need an example on the differences, because in so far as statistical entropy is entropy - it's ultimately describing an information function (and the lean in quantum mechanics these days is that information is describing physical properties as well - hence holography and the blackhole information paradox).

The heat-deathed universe for example would be the ultimate compressible information: 1 measurable state, across all space, for the rest of infinite time.

Re: It took me 10 years to understand entropy

#270

Post by hatware »

jwilk wrote: post #104
hatware wrote: post #40

Google freewall? Guess I won't read this article...

What do you mean?

It is required that I log-in to the article using my google or facebook account to read it.

Garbage. Your article isn't worth reading if I need to jump through hoops to get there.

Post reply on HN