Entropy can be used to understand systems

#1

Post by acgan » Sun, Sep 22, 2019, 9:32 PM UTC

Entropy can be used to understand systems
acgan.sh

Re: Entropy can be used to understand systems

#2

Post by westurner » Sun, Sep 22, 2019, 9:47 PM UTC

Maximum entropy: https://en.wikipedia.org/wiki/Maximum_entropy

Here's a quote of a tweet about a (my own): comment on a schema:BlogPost: https://twitter.com/westurner/status/1048125281146421249:

> “When Bayes, Ockham, and Shannon come together to define machine learning” https://towardsdatascience.com/when-bayes-ockham-and-shannon...

> Comment: "How does this relate to the Principle of Maximum Entropy? How does Minimum Description Length relate to Kolmogorov Complexity?"

Re: Entropy can be used to understand systems

#3

Post by acgan » Sun, Sep 22, 2019, 9:55 PM UTC

westurner wrote: post #2

Maximum entropy: https://en.wikipedia.org/wiki/Maximum_entropy Here's a quote of a tweet about a (my own): comment on a schema:BlogPost: https://twitter.com/westurner/status/1048125281146421249 : > “When Bayes, Ockham, and Shannon come together to define machine learning” https://towardsdatascience.com/when-bayes-ockham-and-shannon... > Comment: "How does this relate to the Principle of Maximum Entropy? How does Mini…

Thanks for sharing! Despite the fact that Shannon's "A Mathematical Theory of Communication" is so accessible, I find that most in our field (stats/ML) don't often think through information-theoretic tools in a "first principles way."

Yes, KL divergences show up everywhere, but they are not derived from scratch often enough. Maybe I'm stifled by my campus bubble though :)