An alternative construction of Shannon entropy

#1

Post by rkp8000 » Wed, Nov 13, 2024, 4:45 PM UTC

An alternative construction of Shannon entropy
rkp.science

Re: An alternative construction of Shannon entropy

#2

Post by xigoi » Sun, Nov 17, 2024, 9:34 PM UTC

The site is unreadable on mobile because it disables overflow on the equations (which it shows as images, even though it’s 2024 and all modern browsers support MathML).

Re: An alternative construction of Shannon entropy

#3

Post by kgwgk » Sun, Nov 17, 2024, 9:59 PM UTC

Re: An alternative construction of Shannon entropy

#4

Post by masfuerte » Sun, Nov 17, 2024, 11:28 PM UTC

xigoi wrote: post #2

The site is unreadable on mobile because it disables overflow on the equations (which it shows as images, even though it’s 2024 and all modern browsers support MathML).

Today I learned! I'd missed the news that MathML was back in Chrome. I've been publishing web pages using MathML for years, along with a note that the equations don't render in Chrome. I can finally remove the note.

Re: An alternative construction of Shannon entropy

#5

Post by ivansavz » Mon, Nov 18, 2024, 12:33 AM UTC

Nice!

The key step of the derivation is counting the "number of ways" to get the histogram with bar heights L1, L2, ... Ln for a total of L observations.

I had to think a bit why the provided formula is true:

   choose(L,L1) * choose(L-L1,L2) * ... * choose(Ln,Ln)

The story I came up with for the first term, is that in the sequence of lenght L, you need to choose L1 locations that will get the symbol x1, so there are choose(L,L1) ways to do that. Next you have L-L1 remaining spots to fill, and L2 of those need to have the symbol x2, hence the choose(L-L1,L2) term, etc.

Re: An alternative construction of Shannon entropy

#6

Post by canjobear » Mon, Nov 18, 2024, 1:28 AM UTC

Interestingly this has a rather frequentist flavor. The probabilities end up coming from frequency ratios in very large samples.

Re: An alternative construction of Shannon entropy

#7

Post by IdealeZahlen » Mon, Nov 18, 2024, 3:24 AM UTC

Calling this 'alternative' construction seems like coming full circle since this line of combinatorial argument is how Boltzmann came up with his H-function in the first place, which inspired Shannon's entropy.

Re: An alternative construction of Shannon entropy

#8

Post by ziofill » Mon, Nov 18, 2024, 3:33 AM UTC

Eh ok, but the trick is then taking the limit for L->infty and use Stirling’s approx which is what Shannon did

Re: An alternative construction of Shannon entropy

#9

Post by mturmon » Mon, Nov 18, 2024, 8:50 AM UTC

This is what I learned as the “theory of types” from Cover and Thomas, chapter 11, from original work by Imre Csiszar. See just under “theorem 6” in

https://web.stanford.edu/class/ee376a/files/2017-18/lecture_...

The key (which is not in OP) is not the construction of E log(p), but in being able to prove that the “typical set” exists (with arbitrarily high probability), and that the entropy is its size.

Re: An alternative construction of Shannon entropy

#10

Post by derbOac » Mon, Nov 18, 2024, 3:01 PM UTC

canjobear wrote: post #6

Interestingly this has a rather frequentist flavor. The probabilities end up coming from frequency ratios in very large samples.

That line of inferential philosophy is very Jaynesian, for those who are interested.