21–30 of 221 posts
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…
Your This Week's Finds were a hugely enjoyable part of my undergraduate education and beyond.
Thank you again.
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…
That covers one and a half of the twelve points he discusses.
Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition…
All definitions of entropy stem from one central, universal definition: Entropy is the amount of energy unable to be used for useful work. Or better put grammatically: entropy describes the effect that not all energy consumed can be used for work.
Earlier quoted context omitted.
That covers one and a half of the twelve points he discusses.
Correct! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.
I concede that it was much shorter though. Well done!
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…
A different explanation (based on macro- and micro-states) makes it intuitively obvious why entropy is non-decreasing with time or, with a little more depth, what entropy has to do with temperature.
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…
Shannon's entropy is a property of the source-channel-receiver system.
Earlier quoted context omitted.
All definitions of entropy stem from one central, universal definition: Entropy is the amount of energy unable to be used for useful work. Or better put grammatically: entropy describes the effect that not all energy consumed can be used for work.
There's a good case to be made that the information-theoretic definition of entropy is the most fundamental one, and the version that shows up in physics is just that concept as applied to physics.
Entropy is the distribution of potential over negative potential. This could be said "the distribution of what ever may be over the surface area of where it may be." This is erroneously taught in conventional information theory as "the number of configurations in a system" or the available information that has yet to be retrieved. Entropy includes the unforseen, and out of scope. Entropy is merely the predisposition…
This answer is as confident as it's wrong and full of gibberish. Entropy is not a "distribution”, it's a functional that maps a probability distribution to a scalar value, i.e. a single number. It's the mean log-probability of a distribution. It's an elementary statistical concept, independent of physical concepts like “pressure”, “potential”, and so on.
Distribution of potential over negative potential. Negative potential is the "surface area", and available potential distributes itself "geometrically". All this is iterative obviously, some periodicity set by universal speed limit.
It really doesn't sound like you disagree with me.
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…