Maximum likelihood estimation and loss functions

To bring things full circle: the cross-entropy loss is the KL divergence. So intuitively, when you're minimizing cross-entropy loss, you're trying to minimize the "divergence" between the true distribution and your model distribution.

This intuition really helped me understand CE loss.

The next step is ELBO — the evidence lower bound.

This reminds me that David Mackay’s book and his lectures are so excellent on these topics.

This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. It's totally practical to learn this stuff on your own from videos, books, and PDFs. First you need to get a solid conceptual grasp on probability distributions and their associated concepts like random variables, conditional probability, and joint probability. Then you'll be ready to learn some mathematical statistics and follow along with all the notation.

Note that you don't need to go deep into measure-theoretic probability or any of that stuff that requires more advanced prior education in math.

One initial thing to understand is that the probability mass/density functions that you get taught in connection with standard probability distributions (binomial, Normal, etc) are functions of the data values: you put in a data value and the function outputs a probability (density), for some fixed parameter values.

At first glance likelihood functions might look the same, but you have to think of them as functions of the parameters; it's the data that's fixed now (it's whatever you observed in your experiment). Once that's clear, the calculus starts to makes sense -- using the derivative of the likelihood function w.r.t. the parameters to find points in parameter space that are local maxima (or directions that are uphill in parameter space etc).

So given a model with unknown parameters, the data set you observe gives rise to a particular likelihood function, in other words the data set gives rise to a surface over your parameter space that you can explore for maxima. Regions of parameter space where your model gives a high probability to your observed data are considered to be regions of parameter space that your data suggests might describe how reality actually is. Of course, that's not taking into account your prior beliefs about which regions of parameter space are plausible, or whether the model was a good choice in the first place, or whether you've got enough data, etc.