Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

151–160 of 221 posts

Re: What Is Entropy?

#151

Post by im3w1l »

enugu wrote: post #119
kergonath wrote: post #113

Earlier quoted context omitted.

It is a problem because all three come with a bagage. Almost none of the things discussed in this thread are invalid when discussing actual physical entropy even though the equations are superficially similar. And then there are lots of people being confidently wrong because they assume that it’s just one concept. It really is not.

Don't see how the connection is superficial. Even the classical macroscopic definition of entropy as ΔS=∫TdQ can be derived from the information theory perspective as Baez shows in article(using entropy maximizing distributions and Lagrange multipliers). If you have a more specific critique, it would be good to discuss.

In classical physics there is no real objective randomness. Particles have a defined position and momentum and those evolve deterministically. If you somehow learned these then the shannon entropy is zero. If entropy is zero then all kinds of things break down.

So now you are forced to consider e.g. temperature an impossibility without quantum-derived randomness, even though temperature does not really seem to be a quantum thing.

Re: What Is Entropy?

#152

Post by kgwgk »

prof-dr-ir wrote: post #149
kgwgk wrote: post #137

Earlier quoted context omitted.

Ok, I don’t think we disagree. But knowing that entropy is a property of a distribution given by that equation is far from “being it” as a definition of the concept of entropy in physics. Anyway, it seems that - like many others - I just misunderstood the “little need for all the mystery” remark.

> is far from “being it” as a definition of the concept of entropy in physics. I simply do not understand why you say this. Entropy in physics is defined using exactly the same equation. The only thing I need to add is the choice of probability distribution (i.e. the choice of ensemble). I really do not see a better "definition of the concept of entropy in physics". (For quantum systems one can nitpick a bit about de…

I’d say that the concept of entropy “in physics” is about (even better: starts with) the choice of a probability distribution. Without that you have just a number associated with each probability distribution - distributions without any physical meaning so those numbers won’t have any physical meaning either.

But that’s fine, I accept that you may think that it’s just a little detail.

(Quantum mechanics has no mystery either.

ih/2pi dA/dt = AH - HA

That’s it. The only thing one needs to add is a choice of operators.)

Re: What Is Entropy?

#153

Post by kgwgk »

im3w1l wrote: post #151
enugu wrote: post #119

Earlier quoted context omitted.

Don't see how the connection is superficial. Even the classical macroscopic definition of entropy as ΔS=∫TdQ can be derived from the information theory perspective as Baez shows in article(using entropy maximizing distributions and Lagrange multipliers). If you have a more specific critique, it would be good to discuss.

In classical physics there is no real objective randomness. Particles have a defined position and momentum and those evolve deterministically. If you somehow learned these then the shannon entropy is zero. If entropy is zero then all kinds of things break down. So now you are forced to consider e.g. temperature an impossibility without quantum-derived randomness, even though temperature does not really seem to be a q…

> Particles have a defined position and momentum

Which we don’t know precisely. Entropy is about not knowing.

> If you somehow learned these then the shannon entropy is zero.

Minus infinity. Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space. (You need an appropriate extension of Shannon’s entropy to continuous distributions.)

> So now you are forced to consider e.g. temperature an impossibility without quantum-derived randomness

Or you may study statistical mechanics :-)

Re: What Is Entropy?

#154

Post by xdavidliu »

kmeisthax wrote: post #93
waveBidder wrote: post #65

Earlier quoted context omitted.

that's actually the normal view, with saying both info and stat mech entropy are the same is an outlier, most popularized by Jaynes.

If information-theoretical and statistical mechanics entropies are NOT the same (or at least, deeply connected) then what stops us from having a little guy[0] sort all the particles in a gas to extract more energy from them? [0] https://en.wikipedia.org/wiki/Maxwell%27s_demon

Sounds like a non-sequitur to me; what are you implying about the Maxwell's demon thought experiment vs the comparison between Shannon and stat-mech entropy?

Re: What Is Entropy?

#156

Post by kergonath »

kgwgk wrote: post #153
im3w1l wrote: post #151

Earlier quoted context omitted.

In classical physics there is no real objective randomness. Particles have a defined position and momentum and those evolve deterministically. If you somehow learned these then the shannon entropy is zero. If entropy is zero then all kinds of things break down. So now you are forced to consider e.g. temperature an impossibility without quantum-derived randomness, even though temperature does not really seem to be a q…

> Particles have a defined position and momentum Which we don’t know precisely. Entropy is about not knowing. > If you somehow learned these then the shannon entropy is zero. Minus infinity. Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space. (You need an appropriate extension of Shannon’s entropy to continuous distributions.) > So now you are forced to consider e…

> Which we don’t know precisely. Entropy is about not knowing.

No, it is not about not knowing. This is an instance of the intuition from Shannon’s entropy does not translate to statistical Physics.

It is about the number of possible microstates, which is completely different. In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change of a material sample and work out its entropy without knowing a thing about its structure.

> Minus infinity. Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space.

No, 0. In this case, there is a single state with p=1 and and S = - k Σ p ln(p) = 0.

This is the same if you consider the phase space because then it is reduced to a single point (you need a bit of distribution theory to prove it rigorously but it is somewhat intuitive).

The probability p of an microstate is always between 0 and 1, therefore p ln(p) is always negative and S is always positive.

You get the same using Boltzmann’s approach, in which case Ω = 1 and S = k ln(Ω) is also 0.

> (You need an appropriate extension of Shannon’s entropy to continuous distributions.)

Gibbs’ entropy.

> Or you may study statistical mechanics

Indeed.

Re: What Is Entropy?

#157

Post by ccosm »

"I have largely avoided the second law of thermodynamics, which says that entropy always increases. While fascinating, this is so problematic that a good explanation would require another book!"

For those interested I am currently reading "Entropy Demystified" by Arieh Ben-Naim which tackles this side of things from much the same direction.

Re: What Is Entropy?

#158

Post by bubblyworld »

_hark wrote: post #150
bubblyworld wrote: post #104

Earlier quoted context omitted.

Thanks, that's an interesting perspective. It also highlights one of the weak points in the concept, I think, which is that this is only a tool for updating beliefs to the extent that the underlying probability space ("ontology" in this analogy) can actually "model" the phenomenon correctly! It doesn't seem to shed much light on when or how you could update the underlying probability space itself (or when to change y…

I think what you're getting at is the construction of the sample space - the space of outcomes over which we define the probability measure (e.g. {H,T} for a coin, or {1,2,3,4,5,6} for a die). Let's consider two possibilities: 1. Our sample space is "incomplete" 2. Our sample space is too "coarse" Let's discuss 1 first. Imagine I have a special die that has a hidden binary state which I can control, which forces the…

Right, this is exactly what I'm getting at - learning a distribution over a fixed sample space can be done with Bayesian methods, or entropy-based methods like the OP suggested, but I'm wondering if there are methods that can automatically adjust the sample space as well.

For well-defined mathematical problems like dice rolling and fixed classical mechanics scenarios and such, you don't need this I guess, but for any real-world problem I imagine half the problem is figuring out a good sample space to begin with. This kind of thing must have been studied already, I just don't know what to look for!

There are some analogies to algorithms like NEAT, which automatically evolves a neural network architecture while training. But that's obviously a very different context.

Re: What Is Entropy?

#159

Post by kgwgk »

kergonath wrote: post #156
kgwgk wrote: post #153

Earlier quoted context omitted.

> Particles have a defined position and momentum Which we don’t know precisely. Entropy is about not knowing. > If you somehow learned these then the shannon entropy is zero. Minus infinity. Entropy in classical statistical mechanics is proportional to the logarithm of the volume in phase space. (You need an appropriate extension of Shannon’s entropy to continuous distributions.) > So now you are forced to consider e…

> Which we don’t know precisely. Entropy is about not knowing. No, it is not about not knowing. This is an instance of the intuition from Shannon’s entropy does not translate to statistical Physics. It is about the number of possible microstates, which is completely different. In Physics, entropy is a property of a bit of matter, it is not related to the observer or their knowledge. We can measure the enthalpy change…

> possible microstates

Conditional on the known macrostate. Because we don’t know the precise microstate - only which microstates are possible.

If your reasoning is that « experimental entropy can be measured so it’s not about that » then it’s not about macrostates and microstates either!

Re: What Is Entropy?

#160

Post by [deleted] »

financltravsty wrote: post #142
canjobear wrote: post #71

Earlier quoted context omitted.

Some probability distributions are objective. The probability that my random number generator gives me a certain number is given by a certain formula. Describing it with another distribution would be wrong. Another example, if you have an electron in a superposition of half spin-up and half spin-down, then the probability to measure up is objectively 50%. Another example, GPT-2 is a probability distribution on sequen…

The probability distribution is subjective for both parts -- because it, once again, depends on the observer observing the events in order to build a probability distribution. E.g. your random number generator generates 1, 5, 7, 8, 3 when you run it. It generates 4, 8, 8, 2, 5 when I run it. I.e. we have received different information about the random number generator to build our subjective probability distributions…

[deleted]
Post reply on HN