61–70 of 123 posts
Here it is explained at length: "An Intuitive Explanation of the Information Entropy of a Random Variable, Or: How to Play Twenty Questions": http://danielwilkerson.com/entropy.html
I'm not sure I understand the distinction between "high-entropy macrostate" and "order". Aren't macrostates just as subjective as order? Let's say my friend's password is 6dVcOgm8. If we have a system whose microstate consists of an arbitrary string of alphanumeric characters, and the system arranges itself in the configuration 6dVcOgm8, then I would describe the macrostate as "random" and "disordered". However, if m…
If you define your macrostates using subjective terms (e.g. "a string that's meaningful to me" or "a string that looks ordered to me") then yeah, your entropy calculations will be subjective.
One thing that helped me was the realization that, at least as used in the context of information theory, entropy is a property of an individual (typically the person receiving a message) and NOT purely of the system or message itself. > entropy quantifies uncertainty This sums it up. Uncertainty is the property of a person and not a system/message. That uncertainty is a function of both a person's model of a system/…
Earlier quoted context omitted.
> If we're calculating the entropy of dice rolls (where the outcome is the 'message'), and I know the dice are loaded but you don't, my entropy will be lower than yours. That's got nothing to do with entropy being subjective. If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion.
> If 2 people are calculating any property and one of them is making a false assumption, they'll end up with a different (false) conclusion. This implies that there is an objectively true conclusion. The true probability is objective.
I'm not sure I understand the distinction between "high-entropy macrostate" and "order". Aren't macrostates just as subjective as order? Let's say my friend's password is 6dVcOgm8. If we have a system whose microstate consists of an arbitrary string of alphanumeric characters, and the system arranges itself in the configuration 6dVcOgm8, then I would describe the macrostate as "random" and "disordered". However, if m…
I think you need to rigorously define your macrostates. If your two states are "my friend's password" and "not my friend's password" then the macrostates are perfectly objective. You don't know what macrostate the system is in, but that doesn't change the fact that the system is objectively in one of those two macrostates. If you define your macrostates using subjective terms (e.g. "a string that's meaningful to me"…
In one case you're looking at the system as "alphanumeric string of length N." In another, the system is that plus something like "my friend's opinion on the string".
Also, as the article says, using "entropy" to mean "order" is not a good practice. "Order" is a subjective concept, and some systems (like oil and water separating) look more "ordered" but still have higher entropy, because there is more going on energetically than we can observe.
Information and statistical explanations of entropy are very easy. The real question is, what does entropy mean in the original context that it was introduced in, before those later explanations?
Nitpick in the article conclusion: >Heat flows from hot to cold because the number of ways in which the system can be non-uniform in temperature is much lower than the number of ways it can be uniform in temperature ... Should probably say "thermal energy" instead of "temperature" if we want to be really precise with our thermodynamics terms. Temperature is not a direct measure of energy, rather it is an extensive pr…
Nitpick of the nitpick... Temperature is actually an intensive quantity, i.e. combining two subsystems with the same temperature yields a bigger system with the same temperature, not twice bigger.
I define both concepts fundamentally in relation to priors and possibilities:
- Entropy is the relationship between priors and ANY possibility, relative to the entire space of possibilities.
- Probability is the relationship between priors and a SPECIFIC possibility, relative to the entire space of possibilities.
The framing of priors and possibilities shows why entropy appears differently across disciplines like statistical mechanics and information theory. Entropy is not merely observer-dependent, but prior-dependent. Including priors not held by any specific observer but embedded in the framework itself. This helps resolve the apparent contradiction between objective and subjective interpretations of entropy.
It also defines possibilities as constraints imposed on an otherwise unrestricted reality. This framing unifies how possibility spaces are defined across frameworks.
[1]: https://buttondown.com/themeaninggap/archive/a-unified-persp...
Earlier quoted context omitted.
Probability is subjective though, because macrostates are subjective. The notion of probability relies on the notion of repeatability: if you repeat a coin flip infinite times, what proportion of outcomes will be heads, etc. But if you actually repeated the toss exactly the same way every time, say with a finely-tuned coin-flipping machine in a perfectly still environment, you would always get the same result. We say…
Probability is a bunch of numbers that add to 1. Sometimes you can use this to represent subjective beliefs. Sometimes you can use it to represent objectively existing probability distributions. For example, an LLM is a probability distribution on a following token given previous tokens. If two "observers" disagree about an LLM's probability assigned to some token, then only at most one of them can be correct. So the…
The observer who knows the implementation in detail and the state of the pseudo-random number generator can predict the next token with certainty. (Or almost certainty, if we consider flip-switching cosmic rays, etc.)
Anyone else notice how the entropy in the 1000 bouncing balls simulation goes down at some point, thereby violating the second law of thermodynamics? :)
Over long enough scales there is no conservation of energy because the universe does not have temporal symmetry.
https://math.ucr.edu/home/baez/physics/Relativity/GR/energy_...
“Is Energy Conserved in General Relativity?”
“In special cases, yes. In general, it depends on what you mean by "energy", and what you mean by "conserved".”