Cross Entropy

Post by pizza » Wed, May 23, 2018, 7:54 PM UTC

tziki wrote: post #10

banachtarski wrote: post #8

"Cross-entropy is defined as the difference between the following two probability distributions" Huh? No this is a mathematically imprecise statement (and not correct either). Most explanations use references to information theory, where a perfect knowledge of the desired probability distribution leads to a perfect allocation of bits in a binary encoding. The entropy is the expected number of bits when this allocatio…

> The entropy is the expected number of bits when this allocation is done using the incorrect distribution Is there any source that would derive and/or explain this more in-depth? I've been trying to develop an intuition for this, but haven't come across a good explanation.

Post by banachtarski » Wed, May 23, 2018, 9:07 PM UTC

tziki wrote: post #10

banachtarski wrote: post #8

"Cross-entropy is defined as the difference between the following two probability distributions" Huh? No this is a mathematically imprecise statement (and not correct either). Most explanations use references to information theory, where a perfect knowledge of the desired probability distribution leads to a perfect allocation of bits in a binary encoding. The entropy is the expected number of bits when this allocatio…

> The entropy is the expected number of bits when this allocation is done using the incorrect distribution Is there any source that would derive and/or explain this more in-depth? I've been trying to develop an intuition for this, but haven't come across a good explanation.