21–29 of 29 posts
https://lospino.so/statistics/jensen-shannon-divergence/
Feedback welcome both from initiates (on helpfulness) and experts (on correctness)!
There is so much I don't understand
Why not use this instead of KL in reinforcement learning?
In practice, which divergence you use doesn't seem to be very important. The KL is the one with the most theoretic foundation though, i.e. will work with infinite data. The important aspect seems to be that neural networks are Lipschitz bound, and that that is the most important constraint preventing collapse.
Why not use this instead of KL in reinforcement learning?
JSD is just symmetrized KL, it's the forward KL + reverse KL. In reinforcement learning, usually what we want is to find the optimal action, i.e. action that maximizes the reward, this translates to the so-called "mode-seeking" optimization, which is the reverse KL.
One way to interpret JSD(P, Q): Associate the distributions P and Q with two target classes, respectively. Pick a target class based on a fair coin flip. Then sample either from distribution P or distribution Q, depending on the outcome of the coin flip. The JSD is the mutual information between the resulting mixture distribution and the target class.
Alternative intuition: Suppose we want to measure the correlation between a feature X and a binary target class Y. We have a tabular data set with two columns X and Y, whose rows correspond to individual samples. JSD is the mutual information between the feature X and the target class Y, but after we resample our data (rows) to ensure that we have a balanced representation of the target class Y. If we measure the JSD in bits, the quantity 2^(JSD-1) is the fraction of times X correctly predicts Y, assuming balanced classes.
There is so much I don't understand
Why not use this instead of KL in reinforcement learning?
Calculating the JSD could be more difficult, the expression uses a mixture between the 'true' and 'fitted' distribution. You can still simulate this, but half the time you'd be fitting the model to itself, and I just don't see why that would be useful.
I think the JSD is most useful when you need an actual metric, but as long as you have a fitted and target distribution the KL divergence is a natural fit since you can interpret the result as information loss.
There is so much I don't understand
Every time I end up on wikipedia page for some math or CS term I just give up on reading and search for other source, any at all. I know it is supposed to be an encyclopedia, and I am sure definitions are technically correct but it just isn't what most people need. I remember wikibooks project tried to bridge that gap but it never got popular enough. I guess it is just easier to compose short notes compared to writin…