Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kullback–Leibler divergence

en.wikipedia.org

Post reply on HN
Print view View original

1–10 of 77 posts

Re: Kullback–Leibler divergence

#2

Post by max_ »

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is.

Could someone give me a simple explanation as to what it's is.

And also, what practical use cases does it have?

Re: Kullback–Leibler divergence

#4

Post by ljlolel »

max_ wrote: post #2

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is. Could someone give me a simple explanation as to what it's is. And also, what practical use cases does it have?

It just describes how two probability distributions are different. If they are the same then it’s 0.

Example: an LLM gives a probability distribution of the next word. If it is perfectly accurate at predicting the next word then divergence is 0 (100% probability on the actual next word). If it is slightly off or unsure then the divergence goes up.

Re: Kullback–Leibler divergence

#5

Post by eob »

max_ wrote: post #2

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is. Could someone give me a simple explanation as to what it's is. And also, what practical use cases does it have?

As for practical use cases, one is to find an approximate optimization to a function

- You want to find the min/max of some probability distribution P(x)

- P(x) is too complicated to find a closed-form min, but you can draw samples from it.

- So instead, you carefully construct some OTHER probability distribution Q(x|θ) that you claim is structurally similar "enough" to P(x), parameterized by θ.

- Now you find the theta which minimizes the KL divergence KL(P(x) || Q(x|θ)), which is equivalent to delivering you the parameters of θ to Q(x|θ) that make it [approximately] "most" similar to P(x) without ever having minimized P(x)

It was a trick that came up a lot when AI consisted of giant Bayesian plate models for each specific task that you had to hand-optimize.

Re: Kullback–Leibler divergence

#6

Post by leourbina »

max_ wrote: post #2

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is. Could someone give me a simple explanation as to what it's is. And also, what practical use cases does it have?

One intuition is that KL-divergence represents a sort of “distance” between probability distributions. However, this isn’t quite right as it doesn’t satisfy some basic properties a real distance (a norm) would satisfy, including the fact that it isn’t symmetric: KL(Q, P) != KL(P,Q), and it does not satisfy the triangle inequality. Nonetheless, KL(P,Q) gives you a good idea of how “far” is P is from Q: in the context of encoding, if you wanted to come up with an ideal encoding of symbols coming from P, but you guessed Q as the distribution of these symbols, then KL(P, Q) is the extra number of bits you’d have to use. One nice property is that in the case that KL(P,Q) = 0, P and Q are equal (almost everywhere, which for most applications is irrelevant). This makes it useful in the ML context as you can minimize KL divergence and know that the resulting “guessed” distribution is getting closer to the data distribution you’re trying to guess using some parametrized function (an NN).

Re: Kullback–Leibler divergence

#7

Post by nonotmenonono »

max_ wrote: post #2

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is. Could someone give me a simple explanation as to what it's is. And also, what practical use cases does it have?

"The content of an information itself, maybe a signal, random variable, or event",

"...defined through a negative logarithm of probability",

"...to model a given outcome", occurred.

P-:

Re: Kullback–Leibler divergence

#8

Post by golwengaud »

I found https://www.lesswrong.com/posts/no5jDTut5Byjqb4j5/six-and-a-... very helpful for getting intuition for what the K-L divergence is and why it's useful. The six intuitions:

    1. Expected surprise
    2. Hypothesis testing
    3. MLEs
    4. Suboptimal coding
    5a. Gambling games -- beating the house
    5b. Gambling games -- gaming the lottery
    6. Bregman divergence

Re: Kullback–Leibler divergence

#9

Post by mrv_asura »

I learnt about KL Divergence recently and it was pretty cool to know that cross-entropy loss originated from KL Divergence. But could someone give me the cases where it is preferred to use Mean-squared Error loss vs Cross-entropy loss? Is there any merits or demerits of using either?

Re: Kullback–Leibler divergence

#10

Post by bjornsing »

max_ wrote: post #2

K-L Divergence is something that Keeps coming up in my research but I still don't understand what it is. Could someone give me a simple explanation as to what it's is. And also, what practical use cases does it have?

I think KL divergence is best understood through the lens of variational inference. Inference in the Bayesian regime is a balancing act between best explaining the data and “keeping it simple”, by staying close to the prior. KL divergence is the (only) divergence measure that places just the right amount of weight on the prior to make variational inference proper Bayesian inference. I’ve written about it here: http://www.openias.org/variational-coin-toss
Post reply on HN