Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

jasonfantl.com

Post reply on HN
Print view View original

41–50 of 123 posts

Re: What Is Entropy?

#41

Post by at_a_remove »

alganet wrote: post #6
soulofmischief wrote: post #4

Earlier quoted context omitted.

Maybe you're confused by entropy? It's pretty well established in different domains. There are multiple ways to look at the same phenomenon, because it's ubiquitous and generalized across systems. It comes down to information and uncertainty. The article in question does attempt to explain all of this if you read it.

Maybe I am. The part of thr article on information theory is more about mathematics than software. I don't deny there could be some generalization there. The problem I see is that this could slip to measure human actions, which are also source of uncertainty, but although the words fit, in this particular case I think associating it with classical entropy does more harm than good. https://en.m.wikipedia.org/wiki/Soft…

You are being fazed by two different, annoying things.

Even in physics itself, the word "mass" has multiple contexts (inertial, gravitational, and conversion to energy) in which it is used. Einstein made quite a lot of hay out of relating the three. "Entropy," too, has multiple contexts.

The second thing possibly tripping you up is the tendency for scientific terms to be poorly appropriated into a new context, like "theory." You can fight this but it is a losing battle, so I typically just try to set it aside.

Re: What Is Entropy?

#42

Post by tshaddox »

nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question.

When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an observer come into play?

It seems to me that two things in this scenario are true:

1) The fundamental physical interactions (i.e. particle collisions) are all time-reversible, and no observer of any one such interaction would be able to tell which directly time is flowing.

2) The states of the overall system are not time-reversible.

Re: What Is Entropy?

#43

Post by oh_my_goodness »

karpathy wrote: post #25

What I never fully understood is that there is some implicit assumption about the dynamics of the system. So what that there are more microstates of some macrostate as far as counting is concerned? We also have to make assumptions about the dynamics, and in particular about some property that encourages mixing.

In equilibrium we don't have to make an assumption about the dynamics or the mixing. We just expect to see the most probable state when we measure.

It's interesting to try to show that the time average equals the ensemble average. It's very cool to think about the dynamics. That stuff must be happening. But those extra ideas aren't necessary for applying the equilibrium theory.

Re: What Is Entropy?

#44

Post by johan_felisaz »

anon84873628 wrote: post #37

Nitpick in the article conclusion: >Heat flows from hot to cold because the number of ways in which the system can be non-uniform in temperature is much lower than the number of ways it can be uniform in temperature ... Should probably say "thermal energy" instead of "temperature" if we want to be really precise with our thermodynamics terms. Temperature is not a direct measure of energy, rather it is an extensive pr…

Nitpick of the nitpick... Temperature is actually an intensive quantity, i.e. combining two subsystems with the same temperature yields a bigger system with the same temperature, not twice bigger.

Re: What Is Entropy?

#45

Post by alganet »

at_a_remove wrote: post #41
alganet wrote: post #6

Earlier quoted context omitted.

Maybe I am. The part of thr article on information theory is more about mathematics than software. I don't deny there could be some generalization there. The problem I see is that this could slip to measure human actions, which are also source of uncertainty, but although the words fit, in this particular case I think associating it with classical entropy does more harm than good. https://en.m.wikipedia.org/wiki/Soft…

You are being fazed by two different, annoying things. Even in physics itself, the word "mass" has multiple contexts (inertial, gravitational, and conversion to energy) in which it is used. Einstein made quite a lot of hay out of relating the three. "Entropy," too, has multiple contexts. The second thing possibly tripping you up is the tendency for scientific terms to be poorly appropriated into a new context, like "…

A borrowed term in another context is not the same as generalization. Read my responses again.

I think my first comment was clear enough about it without needing to step into semantics.

Re: What Is Entropy?

#46

Post by thawtxp »

tshaddox wrote: post #42
nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an obs…

[dead]

Re: What Is Entropy?

#47

Post by [deleted] »

vitus wrote: post #36
tomnicholas1 wrote: post #35

Earlier quoted context omitted.

Isn't that more about enumerating the microstates? The Pauli exclusion principle just ends up forbidding some of the microstates (forbidding a significant fraction of them if you're in the low-temperature regime).

It is about enumerating the microstates, but in a way that takes into account how the particles interact with each other (aka making assumptions about the dynamics). If we didn't take into account any interactions, we'd be unable to do anything with statistical mechanics beyond rederiving the ideal gas law.

[deleted]

Re: What Is Entropy?

#48

Post by hatthew »

I'm not sure I understand the distinction between "high-entropy macrostate" and "order". Aren't macrostates just as subjective as order? Let's say my friend's password is 6dVcOgm8. If we have a system whose microstate consists of an arbitrary string of alphanumeric characters, and the system arranges itself in the configuration 6dVcOgm8, then I would describe the macrostate as "random" and "disordered". However, if my friend sees that configuration, they would describe the macrostate as "my password" and "ordered".

If we see another configuration M2JlH8qc, I would say that the macrostate is the same, it's still "random" and "unordered", and my friend would agree. I say that both macrostates are the same: "random and unordered", and there are many microstates that could be called that, so therefore both are microstates representing the same high-entropy macrostate. However, my friend sees the macrostates as different: one is "my password and ordered", and the other is "random and unordered". There is only one microstate that she would describe as "my password", so from her perspective that's a low-entropy macrostate, while they would agree with me that M2JlH8qc represents a high-entropy macrostate.

So while I agree that "order" is subjective, isn't "how many microstates could result in this macrostate" equally subjective? And then wouldn't it be reasonable to use the words "order" and "disorder" to count (in relative terms) how many microstates could result in the macrostate we subjectively observe?

Re: What Is Entropy?

#49

Post by dynm »

tshaddox wrote: post #42
nihakue wrote: post #20

I'm not in any way qualified to have a take here, but I have one anyway: My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. So for example you could observe that a single coin flip is heads, a…

> My understanding is that entropy is a way of quantifying how many different ways a thing could 'actually be' and yet still 'appear to be' how it is. So it is largely a result of an observer's limited ability to perceive / interrogate the 'true' nature of the system in question. When ice cubes in a glass of water slowly melt, and the temperature of the liquid water decreases, where does the limited ability of an obs…

It's tricky when you think of a continuous system because the "differential entropy" is different (and more subtle) than the "entropy". Even if a system is time-reversible, the "measure" of a set of states can change.

For example: Say I'm at some distance from you, between 0 and 1 km (all equiprobable). Now I switch to being 10x as far away. This is time-reversible, but because the volume of the set of states changed, the differential entropy changes. This is the kind of thing that happens in time-reversible continuous systems that can't happen in time-reversible discrete systems.

Re: What Is Entropy?

#50

Post by voidhorse »

I think this is a pretty good introduction but it gets a little bogged down in the binary encoding assumption, which is an extraneous detail. It does help to know why the logarithm is chosen as a measure of information though regardless of base, once you know that "entropy" is straightforward. I'd agree that much of the difficulty arises from the uninformative name and the various mystique it carries.

To try to expand on the information measure part from a more abstract starting point: Consider a probability distribution, some set of probabilities p. We can consider it as indicating our degree of certainty about what will happen. In an equiprobable distribution, e.g. a fair coin flip (1/2, 1/2) there is no skew either which way, we are admitting that we basically have no reason to suspect any particular outcome. Contrarily, in a split like (1/4, 3/4) we are stating that we are more certain that one particular outcome will happen.

If you wanted to come up with a number to represent the amount of uncertainty, it's clear that the number should be higher the closer the distribution is to being completely equiprobable (1/2, 1/2)—complete lack of certainty about the result, and the number should be smallest when we are 100% certain (0, 1).

This means that the function has to be an order inversion on the probability values—that is I(1) = 0 (no uncertainty). The logarithm, to arbitrary base (selecting a base is just a change of units) has this property under the convention that I(0) = inf (that is, a totally improbable event carries infinite information—after all, an impossibility occurring would in fact be the ultimate surprise).

Entropy is just the average of this function taken over the probability values (multiply each probability in the distribution by the log of the inverse of the probabilities and sum them). In info theory you also usually assume the probabilities are independent, and so the further condition that I(pq) = I(p) + I(q) is also stipulated.

Post reply on HN