What Is Entropy? • Hacker News Board

Re: What Is Entropy?

#101

Post by dkislyuk » Tue, Apr 15, 2025, 12:22 PM UTC

quietbritishjim wrote: post #97

I like the axiomatic definition of entropy. Here's the introduction from Pattern Recognition and Machine Learning by C. Bishop (2006): > The amount of information can be viewed as the ‘degree of surprise’ on learning the value of x. If we are told that a highly improbable event has just occurred, we will have received more information than if we were told that some very likely event has just occurred, and if we knew…

This is a great characterization of self-information. I would add that the `log` term doesn't just conveniently appear to satisfy the additivity axiom, but instead is the exact historical reason why it was invented in the first place. As in, the log function was specifically defined to find a family of functions that satisfied f(xy) = f(x) + f(y).

So, self-information is uniquely defined by (1) assuming that information is a function transform of probability, (2) that no information is transmitted for an event that certainly happens (i.e. f(1) = 0), and (3) independent information is additive. h(x) = -log p(x) is the only set of functions that satisfies all of these properties.

Re: What Is Entropy?

#102

Post by HelloNurse » Tue, Apr 15, 2025, 1:58 PM UTC

dist-epoch wrote: post #85

frank20022 wrote: post #78

Earlier quoted context omitted.

By that definition, the entropy of a game of chess decreases with time because as the game moves on there are less possible legal states. Did I get that right?

Is about subjective knowledge, not objective. So entropy is not related to the number of remaining legal states. If I know the seed of a PRNG, the entropy of the numbers it generates is zero for me. If I don't know the seed, it has very high entropy. https://www.quantamagazine.org/what-is-entropy-a-measure-of-...

Low entropy in chess means effective prediction of future moves, which is far less useful than dedicating equivalent effort to choosing a good next move for ourselves.

Re: What Is Entropy?

#103

Post by tshaddox » Tue, Apr 15, 2025, 2:31 PM UTC

quietbritishjim wrote: post #97

I like the axiomatic definition of entropy. Here's the introduction from Pattern Recognition and Machine Learning by C. Bishop (2006): > The amount of information can be viewed as the ‘degree of surprise’ on learning the value of x. If we are told that a highly improbable event has just occurred, we will have received more information than if we were told that some very likely event has just occurred, and if we knew…

According to my perhaps naive interpretation of that, the "degree of surprise" would depend on at least three things:

1. the laws of nature (i.e. how accurately do the laws of physics permit measuring the system and how determined are future states based on current states)

2. one's present understanding of the laws of nature

3. one's ability to measure the state of a system accurately and compute the predictions in practice

It strikes me as odd to include 2 and 3 in a definition of "entropy."

Re: What Is Entropy?

#104

Post by anon84873628 » Tue, Apr 15, 2025, 3:00 PM UTC

kgwgk wrote: post #73

anon84873628 wrote: post #37

Nitpick in the article conclusion: >Heat flows from hot to cold because the number of ways in which the system can be non-uniform in temperature is much lower than the number of ways it can be uniform in temperature ... Should probably say "thermal energy" instead of "temperature" if we want to be really precise with our thermodynamics terms. Temperature is not a direct measure of energy, rather it is an extensive pr…

I think you used “extensive” in the sense of “defined for the whole system and not locally”. It’s true that thermodynamics is about systems at equilibrium.

I meant to say "intensive" in the physics sense but just brain farted while typing.

Re: What Is Entropy?

#105

Post by im3w1l » Tue, Apr 15, 2025, 3:01 PM UTC

dynm wrote: post #49

tshaddox wrote: post #42

Earlier quoted context omitted.

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an obs…

It's tricky when you think of a continuous system because the "differential entropy" is different (and more subtle) than the "entropy". Even if a system is time-reversible, the "measure" of a set of states can change. For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed…

Isn't that kind of what we want entropy to capture though? If a particle darts off into the distance then in theory it might be time reversible, but in practice it's not so simple. If the particle escapes the gravitational pull, the only way it can come back is if it bumps into some other object and pushes that object away. So things will inevitably spread out more and more creating an arrow of time.

This can then be related to the big bang, and maybe it could be said that we are all living of the negentropy from that event and the subsequent expansion.

Getting different entropy values based on choice of units is a very nasty property though. It kinda hints that there is one canonical correct unit (plank length?)

Re: What Is Entropy?

#106

Post by im3w1l » Tue, Apr 15, 2025, 3:06 PM UTC

As a kid I wanted to invent a perpetuum mobile. From that perspective, entropy is that troublesome property that prevents a perpetuum mobile of the second kind. And any fuzziness or ambiguity in its definition is a glimmer of hope that we may yet find a loop hole.

Re: What Is Entropy?

#107

Post by im3w1l » Tue, Apr 15, 2025, 3:11 PM UTC

So here is an amusing thought experiment I thought of at one point.

Imagine a very high resolution screen. Say a billion by a billion pixels. Each of them can be white, gray or black. What is the lowest entropy possible? Each of the pixels has the same color. How does the screen look? Gray. What is the highest entropy possible? Each pixel has a random color. How does it look from a distance? Gray again.

What does this mean? I have no idea. Maybe nothing.

Also sorry for writing two top level comments, but I just really care about this topic

Re: What Is Entropy?

#108

Post by tmalsburg2 » Tue, Apr 15, 2025, 3:51 PM UTC

tshaddox wrote: post #103

quietbritishjim wrote: post #97

I like the axiomatic definition of entropy. Here's the introduction from Pattern Recognition and Machine Learning by C. Bishop (2006): > The amount of information can be viewed as the ‘degree of surprise’ on learning the value of x. If we are told that a highly improbable event has just occurred, we will have received more information than if we were told that some very likely event has just occurred, and if we knew…

According to my perhaps naive interpretation of that, the "degree of surprise" would depend on at least three things: 1. the laws of nature (i.e. how accurately do the laws of physics permit measuring the system and how determined are future states based on current states) 2. one's present understanding of the laws of nature 3. one's ability to measure the state of a system accurately and compute the predictions in p…

OP is talking about information entropy. Nature isn't relevant there.

Re: What Is Entropy?

#109

Post by kgwgk » Tue, Apr 15, 2025, 3:56 PM UTC

anon84873628 wrote: post #104

kgwgk wrote: post #73

Earlier quoted context omitted.

I think you used “extensive” in the sense of “defined for the whole system and not locally”. It’s true that thermodynamics is about systems at equilibrium.

I meant to say "intensive" in the physics sense but just brain farted while typing.

Ah, then I don’t see what’s wrong with “the number of ways in which the system can be non-uniform in temperature is much lower than the number of ways it can be uniform in temperature”. In equilibrium one doesn’t have a gradient of temperature because “…” indeed.

Re: What Is Entropy?

#110

Post by infogulch » Tue, Apr 15, 2025, 3:56 PM UTC

frank20022 wrote: post #78

infogulch wrote: post #54

Earlier quoted context omitted.

Half a year after that talk Wolfram appeared on a popular podcast [1] to discuss his book on the Second Law of Thermodynamics [2]. That discussion contained the best one-sentence description of entropy I've ever heard: > Entropy is the logarithm of the number of states that are consistent with what you know about a system. [1]: Mystery of Entropy FINALLY Solved After 50 Years? (Stephen Wolfram) - Machine Learning Str…

By that definition, the entropy of a game of chess decreases with time because as the game moves on there are less possible legal states. Did I get that right?

Sure. Lots of games result in a reduction in game state entropy as the game progresses. Many card games could be described as unnecessarily complicated ways to sort a deck, as an example. When analyzing games wrt the Second Law, consider that "the system" is not simply the current game state, but should at least include captured pieces and human choices.