Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

What Is Entropy?

johncarlosbaez.wordpress.com

Post reply on HN
Print view View original

31–40 of 221 posts

Re: What Is Entropy?

#31

Post by illuminant »

ziofill wrote: post #5
illuminant wrote: post #2

Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition…

> Entropy includes the unforseen, and out of scope. Mmh, no it doesn't. You need to define your state space, otherwise it's an undefined quantity.

You are referring to the conceptual device you believe bongs to you and your equations. Entropy creates attraction and repulsion, even causing working bias. We rely upon it for our system functions.

Undefined is uncertainty is entropic.

Re: What Is Entropy?

#32

Post by dist-epoch »

glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

Trivial example: if you know the seed of a pseudo-random number generator, a sequence generated by it has very low entropy.

But if you don't know the seed, the entropy is very high.

Re: What Is Entropy?

#33

Post by petsfed »

dekhn wrote: post #9

I really liked the approach my stat mech teacher used. In nearly all situations, entropy just ends up being the log of the number of ways a system can be arranged ( https://en.wikipedia.org/wiki/Boltzmann%27s_entropy_formula ) although I found it easiest to think in terms of pairs of dice rolls.

And this is what I prefer too, although with the clarification that its the number of ways that a system can be arranged without changing its macroscopic properties.

Its, unfortunately, not very compatible with Shannon's usage in any but the shallowest sense, which is why it stays firmly in the land of physics.

Re: What Is Entropy?

#34

Post by nabla9 »

prof-dr-ir wrote: post #20

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…

Everyone who sees that formula can immediately see that it leads to principle of maximum entropy.

Just like everyone seeing Maxwell's equations can immediately see that you can derive the the speed of light classically.

Oh dear. The joy of explaining the little you know.

Re: What Is Entropy?

#35

Post by dmn322 »

This seems like a great resource for referencing the various definitions. I've tried my hand at developing an intuitive understanding: https://spacechimplives.substack.com/p/observers-and-entropy. TLDR - it's an artifact of the model we're using. In the thermodynamic definition, the energy accounted for in the terms of our model is information. The energy that's not is entropic energy. Hence why it's not "useable" energy, and the process isn't reversible.

Re: What Is Entropy?

#36

Post by prof-dr-ir »

rachofsunshine wrote: post #26
prof-dr-ir wrote: post #20

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…

The problem is that this doesn't get at many of the intuitive properties of entropy. A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.

The above evidently only suffices as a definition, not as an entire course. My point was just that I don't think any other introduction beats this one, especially for a book with the given title.

In particular it has always been my starting point whenever I introduce (the entropy of) macro- and micro-states in my statistical physics course.

Re: What Is Entropy?

#37

Post by mitthrowaway2 »

rachofsunshine wrote: post #30
glial wrote: post #18

I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…

This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).

Entropy is a property of a distribution, but since math does sometimes get applied, we also attach distributions to things (eg. the entropy of a random number generator, the entropy of a gas...). Then when we talk about the entropy of those things, those entropies are indeed subjective, because different subjects will attach different probability distributions to that system depending on their information about that system.

Re: What Is Entropy?

#38

Post by mitthrowaway2 »

prof-dr-ir wrote: post #20

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…

So the only thing you need to know about entropy is that it's a real-valued number you can associate with a probability distribution? And that's it? I disagree. There are several numbers that can be associated with probability distribution, and entropy is an especially useful one, but to understand why entropy is useful, or why you'd use that function instead of a different one, you'd need to know a few more things than just what you've written here.

Re: What Is Entropy?

#40

Post by prof-dr-ir »

nabla9 wrote: post #34
prof-dr-ir wrote: post #20

If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…

Everyone who sees that formula can immediately see that it leads to principle of maximum entropy. Just like everyone seeing Maxwell's equations can immediately see that you can derive the the speed of light classically. Oh dear. The joy of explaining the little you know.

As of this moment there are six other top-level comments which each try to define entropy, and frankly they are all wrong, circular, or incomplete. Clearly the very definition of entropy is confusing, and the definition is what my comment provides.

I never said that all the other properties of entropy are now immediately visible. Instead I think it is the only universal starting point of any reasonable discussion or course on the subject.

And lastly I am frankly getting discouraged by all the dismissive responses. So this will be my last comment for the day, and I will leave you in the careful hands of, say, the six other people who are obviously so extremely knowledgeable about this topic. /s

Post reply on HN