41–50 of 221 posts
Earlier quoted context omitted.
> Entropy includes the unforseen, and out of scope. Mmh, no it doesn't. You need to define your state space, otherwise it's an undefined quantity.
You are referring to the conceptual device you believe bongs to you and your equations. Entropy creates attraction and repulsion, even causing working bias. We rely upon it for our system functions. Undefined is uncertainty is entropic.
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…
> it's a subjective quantity -- a property of the observer, not the observed Shannon's entropy is a property of the source-channel-receiver system.
Entropy is calculated as a function of a probability distribution over possible messages or symbols. The sender might have a distribution P over possible symbols, and the receiver might have another distribution Q over possible symbols. Then the "true" distribution over possible symbols might be another distribution yet, call it R. The mismatch between these is what leads to various inefficiencies in coding, decoding, etc [1]. But both P and Q are beliefs about R -- that is, they are properties of observers.
[1] https://en.wikipedia.org/wiki/Kullback–Leibler_divergence#Co...
PBS Spacetime‘s entropy playlist: https://youtube.com/playlist?list=PLsPUh22kYmNCzNFNDwxIug8q1...
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…
Trivial example: if you know the seed of a pseudo-random number generator, a sequence generated by it has very low entropy. But if you don't know the seed, the entropy is very high.
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…
I felt like I finally understood Shannon entropy when I realized that it's a subjective quantity -- a property of the observer, not the observed. The entropy of a variable X is the amount of information required to drive the observer's uncertainty about the value of X to zero. As a correlate, your uncertainty and mine about the value of the same variable X could be different. This is trivially true, as we could each…
This doesn't really make entropy itself observer dependent. (Shannon) entropy is a property of a distribution. It's just that when you're measuring different observers' beliefs, you're looking at different distributions (which can have different entropies the same way they can have different means, variances, etc).
Earlier quoted context omitted.
Correct! And it took me just one paragraph, not the 18 pages of meandering (and I think confusing) text that it takes the author of the pdf to introduce the same idea.
You didn’t introduce any idea. You said it’s “just a number” and wrote down a formula without any explanation or justification. I concede that it was much shorter though. Well done!
If I would write a book with that title then I would get to the point a bit faster, probably as follows. Entropy is just a number you can associate with a probability distribution. If the distribution is discrete, so you have a set p_i, i = 1..n, which are each positive and sum to 1, then the definition is: S = - sum_i p_i log( p_i ) Mathematically we say that entropy is a real-valued function on the space of probabi…