Live data from Hacker News

What Is Entropy?

jasonfantl.com

51–60 of 123 posts

Re: What Is Entropy?

#51
post #19

Earlier quoted context omitted.

Probability is subjective though, because macrostates are subjective. The notion of probability relies on the notion of repeatability: if you repeat a coin flip infinite times, what proportion of outcomes will be heads, etc. But if you actually repeated the toss exactly the same way every time, say with a finely-tuned coin-flipping machine in a perfectly still environment, you would always get the same result. We say…

Probability is a bunch of numbers that add to 1. Sometimes you can use this to represent subjective beliefs. Sometimes you can use it to represent objectively existing probability distributions. For example, an LLM is a probability distribution on a following token given previous tokens. If two "observers" disagree about an LLM's probability assigned to some token, then only at most one of them can be correct. So the…

We're talking about 2 different things. I agree that probability is objective as long as you've already decided on the definition of the macrostate, but that definition is subjective.

From an LLM's perspective, the macrostate is all the tokens in the context window and nothing more. A different observer may be able to take into account other information, such as the identity and mental state of the author, giving rise to a different distribution. Both of these models can be objectively valid even though they're different, because they rely on different definitions of the macrostate.

It can be hard to wrap your head around this, but try taking it to the extreme. Let's say there's an omniscient being that knows absolutely everything there is to know about every single atom within a system. To that observer, probability does not exist, because every macrostate represents a single microstate. In order for something to be repeated (which is core to the definition of probability), it must start from the exact same microstate, and thus always have the same outcome.

You might think that true randomness exists at the quantum level and that means true omniscience is impossible (and thus irrelevant), but that's not provable and, even if it were true, would not invalidate the general point that probabilities are determined by macrostate definition.

Re: What Is Entropy?

#52
post #49

Earlier quoted context omitted.

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an obs…

It's tricky when you think of a continuous system because the "differential entropy" is different (and more subtle) than the "entropy". Even if a system is time-reversible, the "measure" of a set of states can change. For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed…

I have yet to see differential entropy used successfully (beyond its explicitly constructed-for purpose for calculating channel capacity). Similar to your thought experiment is the issue that the differential entropy value depends on your choice of unit system. Fundamentally the issue is that you cant stick a quantity with units into a transcendental function and get meaningful results out

Re: What Is Entropy?

#53
post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an obs…

The temperature of an object is a macroscopic property basically depending on the kinetic energy of the matter within it, which in a typical cup of water varies substantially from one molecule to the next. If before you could guess a little bit about the kinetic energy of a given water molecule based on whether it is part of the ice or not, after melting and sufficient time to equilibrate the location of a particular molecule gives you no additional information for estimating its velocity.

Re: What Is Entropy?

#54

I don’t see Sean Carroll’s musings mentioned yet, so repeating my previous comment: Entropy got a lot more exciting to me after hearing Sean Carroll talk about it. He has a foundational/philosophical bent and likes to point out that there are competing definitions of entropy set on different philosophical foundations, one of them seemingly observer dependent: - https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 - https…

Half a year after that talk Wolfram appeared on a popular podcast [1] to discuss his book on the Second Law of Thermodynamics [2]. That discussion contained the best one-sentence description of entropy I've ever heard:

> Entropy is the logarithm of the number of states that are consistent with what you know about a system.

[1]: Mystery of Entropy FINALLY Solved After 50 Years? (Stephen Wolfram) - Machine Learning Street Talk Podcast - https://www.youtube.com/watch?v=dkpDjd2nHgo

[2]: The Second Law: Resolving the Mystery of the Second Law of Thermodynamics - https://www.amazon.com/Second-Law-Resolving-Mystery-Thermody...

Re: What Is Entropy?

#55
post #51

Earlier quoted context omitted.

Probability is a bunch of numbers that add to 1. Sometimes you can use this to represent subjective beliefs. Sometimes you can use it to represent objectively existing probability distributions. For example, an LLM is a probability distribution on a following token given previous tokens. If two "observers" disagree about an LLM's probability assigned to some token, then only at most one of them can be correct. So the…

We're talking about 2 different things. I agree that probability is objective as long as you've already decided on the definition of the macrostate, but that definition is subjective. From an LLM's perspective, the macrostate is all the tokens in the context window and nothing more. A different observer may be able to take into account other information, such as the identity and mental state of the author, giving ris…

Suppose you're training a language model by minimizing cross entropy, and the omniscient being is watching. In each step, your model instantiates some probability distribution, whose gradients are computed. That distribution exists, and is not deterministic to the omniscient entity.

Re: What Is Entropy?

#56

Nitpick in the article conclusion: >Heat flows from hot to cold because the number of ways in which the system can be non-uniform in temperature is much lower than the number of ways it can be uniform in temperature ... Should probably say "thermal energy" instead of "temperature" if we want to be really precise with our thermodynamics terms. Temperature is not a direct measure of energy, rather it is an extensive pr…

Nitpick of the nitpick... Temperature is actually an intensive quantity, i.e. combining two subsystems with the same temperature yields a bigger system with the same temperature, not twice bigger.

This is why the "thermodynamic beta" is really useful.

https://en.wikipedia.org/wiki/Thermodynamic_beta

Re: What Is Entropy?

#58

Anyone else notice how the entropy in the 1000 bouncing balls simulation goes down at some point, thereby violating the second law of thermodynamics? :)

Over long enough scales there is no conservation of energy because the universe does not have temporal symmetry.

Re: What Is Entropy?

#59

I don’t see Sean Carroll’s musings mentioned yet, so repeating my previous comment: Entropy got a lot more exciting to me after hearing Sean Carroll talk about it. He has a foundational/philosophical bent and likes to point out that there are competing definitions of entropy set on different philosophical foundations, one of them seemingly observer dependent: - https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 - https…

[deleted]

Re: What Is Entropy?

#60

I don’t see Sean Carroll’s musings mentioned yet, so repeating my previous comment: Entropy got a lot more exciting to me after hearing Sean Carroll talk about it. He has a foundational/philosophical bent and likes to point out that there are competing definitions of entropy set on different philosophical foundations, one of them seemingly observer dependent: - https://youtu.be/x9COqqqsFtc?si=cQkfV5IpLC039Cl5 - https…

By Jeeves, it's rentropy!!

Sean and Stephen are absolutely thoughtful popularizers, but complexity, not entropy, is what they are truly interested in talking about.

Although it doesn't make complexity less scary, here's something Sean's been working on for more than a decade. The paper seems to be more accessible to the layman than he thinks..

https://arxiv.org/abs/1405.6903 https://scottaaronson.blog/?p=762

[When practitioners say "entropy", they mean RELATIVE ENTROPY, which is another can of worms.. rentropy is the one that is observer dependent: "That's Relative as in Relativity". Entropy by itself is simple, blame von Neumann for making it live rent-free]

https://en.wikipedia.org/wiki/Relative_entropy

@nyrikki below hints (too softly, imho) at this:

>You can also approach the property that people often want to communicate when using the term entropy as effective measure 0 sets, null cover, martingales, kolmogorov complexity, compressibility, set shattering, etc...

Post reply on HN