PBS Spacetime‘s entropy playlist: https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1...
What Is Entropy?
11–20 of 221 posts
Re: What Is Entropy?
#12My definition: Entropy is a measure of the accumulation of non-reversible energy transfers.
Side note: All reversible energy transfers involve an increase in potential energy. All non-reversible energy transfers involve a decrease in potential energy.
Re: What Is Entropy?
#13A well known anecdote reported by Shannon: "My greatest concern was what to call it. I thought of calling it 'information,' but the word was overly used, so I decided to call it 'uncertainty.' When I discussed it with John von Neumann, he had a better idea. Von Neumann told me, 'You should call it entropy, for two reasons. In the first place your uncertainty function has been used in statistical mechanics under that…
Von Neumann was the king of kings
Re: What Is Entropy?
#14For information theory, I've always thought of entropy as follows:
"If you had a really smart compression algorithm, how many bits would it take to accurately represent this file?"
i.e., Highly repetitive inputs compress well because they don't have much entropy per bit. Modern compression algorithms are good enough on most data to be used as a reasonable approximation for the true entropy.
Re: What Is Entropy?
#15Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition…
All definitions of entropy stem from one central, universal definition: Entropy is the amount of energy unable to be used for useful work. Or better put grammatically: entropy describes the effect that not all energy consumed can be used for work.
I think what you describe is the application of entropy in the thermodynamic setting, which doesn't apply to "all definitions".
Re: What Is Entropy?
#16My definition: Entropy is a measure of the accumulation of non-reversible energy transfers. Side note: All reversible energy transfers involve an increase in potential energy. All non-reversible energy transfers involve a decrease in potential energy.
That definition doesn't work well because you can have changes in entropy even if no energy is transferred, e.g. by exchanging some other conserved quantity.
The side note is wrong in letter and spirit; turning potential energy into heat is one way for something to be irreversible, but neither of those statements is true.
For example, consider an iron ball being thrown sideways. It hits a pile of sand and stops. The iron ball is not affected structurally, but its kinetic energy is transferred (almost entirely) to heat energy. If the ball is thrown slightly upwards, potential energy increases but the process is still irreversible.
Also, the changes of potential energy in corresponding parts of two Carnot cycles are directionally the same, even if one is ideal (reversible) and one is not (irreversible).
Re: What Is Entropy?
#17Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition…
This answer is as confident as it's wrong and full of gibberish.
Entropy is not a "distribution”, it's a functional that maps a probability distribution to a scalar value, i.e. a single number.
It's the mean log-probability of a distribution.
It's an elementary statistical concept, independent of physical concepts like “pressure”, “potential”, and so on.
Re: What Is Entropy?
#18I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed.
The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each have received different information that about X. H(X) should be H_{observer}(X), or even better, H_{observer, time}(X).
As clear as Shannon's work is in other respects, he glosses over this.
Re: What Is Entropy?
#19My goto source for understanding entropy:
http://philsci-archive.pitt.edu/8592/1/EntropyPaperFinal.pdf
Re: What Is Entropy?
#20If I would write a book with that title then I would get to the point a bit faster, probably as follows.
Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is:
S = - sum_i p_i log( p_i )
Mathematically we say that entropy is a real-valued function on the space of probability distributions. (Elementary exercises: show that S >= 0 and it is maximized on the uniform distribution.)
That is it. I think there is little need for all the mystery.