[deleted]
Jensen–Shannon Divergence
11–20 of 29 posts
Re: Jensen–Shannon Divergence
#12There is so much I don't understand
Re: Jensen–Shannon Divergence
#13It has applications outside of machine learning too! I used symmetric Kullback–Leibler divergence for a project with photon number resolving single photon detectors during my PhD. I used it with an adjacency matrix to split a gaussian mixture model (modelling some data with multivariate gaussians) into a series of clusters.
https://snsphd.online/chapter_04/section_05_results/#photon-...
Re: Jensen–Shannon Divergence
#14Why not use this instead of KL in reinforcement learning?
JSD is just symmetrized KL, it's the forward KL + reverse KL.
In reinforcement learning, usually what we want is to find the optimal action, i.e. action that maximizes the reward, this translates to the so-called "mode-seeking" optimization, which is the reverse KL.
Re: Jensen–Shannon Divergence
#15Earlier quoted context omitted.
I’d like to know what the advantage is over KL divergence. It seems like the important idea is symmetry? Not clear to me why that matters; I’d love to know what application this is used for.
There are many applications. I mainly see it used for detecting drift in datasets for ML models. It has a nice benefit over the KL divergence in the case where the two distributions you're measuring have no overlap (KL won't compute, but JS will just return 0). Also, when taking its square root you get a distance rather than a divergence which allows you to compare it to JSD measurements of other distributions.
> Also, when taking its square root you get a distance
Easy conversion into a distance metric is hugely valuable to making the property amenable to KNN-based dimensionality reduction algos (and I'm sure other things I don't understand, as a non-mathematician)
Here's a library that the creator of UMAP provides (UMAP being a workhorse of dimensional reduction algos), for doing approx nearest neighbor search: https://pynndescent.readthedocs.io/en/latest/api.html#pynnde...
Re: Jensen–Shannon Divergence
#16I thought Jensen Huang was getting a divorce :D
Re: Jensen–Shannon Divergence
#17I thought Jensen Huang was getting a divorce :D
I thought it was some anthropic nvidia breakup
Re: Jensen–Shannon Divergence
#18It has applications outside of machine learning too! I used symmetric Kullback–Leibler divergence for a project with photon number resolving single photon detectors during my PhD. I used it with an adjacency matrix to split a gaussian mixture model (modelling some data with multivariate gaussians) into a series of clusters. https://snsphd.online/chapter_04/section_05_results/#photon-...
[deleted]
Re: Jensen–Shannon Divergence
#19Love me some JSD. Here is a problem most people don't consider with generative modeling (e.g., AI text, image, music, video models): basically all standard pre-training algorithms for generative models (i.e., cross entropy, basically all diffusion/flow formulations) are closer to a Forward KL divergence. In other words, given limited capacity the model will try to stretch itself to cover every mode. This gives you a jack of all trades (lots of knowledge and diversity), but a master of none (you get blurry images and text filled with nonsense).
The real magic in generative modeling comes from the post training process that comes after, which usually (e.g., RLHF) approximates Reverse KL (given limited capacity, try to perfectly cover what you can, but it's fine to drop the rest entirely). This gives amazing results, but is also the cause of AI oddities like the "AI Image Pixar Look", many of the verbal tics of LLMs, and all AI music using the same small set of voices. Jensen-Shannon Divergence sits right in the middle of Forward and Reverse KL and is what many GANs are claimed to approximate. Ideally, it is a better trade-off between diversity and fidelity.
Re: Jensen–Shannon Divergence
#20Currently piloting the use of JSD for a synthetic audience survey application, measuring how closely the synthetic response distribution matches a human panel.
Been knee-deep trying to understand this world, so seeing this on Hacker News today is kind of scary.