Likelihoods aren’t fundamentally small. The center of a normal distribution has high likelihood (e.g. 1000000) if the standard deviation is small or low likelihood if the standard deviation is large (e.g. 1/1000000.) This effect is amplified when you are working with products of likelihoods. They can be infinitesimal or astronomical. Giant likelihoods really surprised me the first time I experienced them but they’re…
Why is everything based on likelihoods even though likelihoods are so small?
61–70 of 103 posts
Re: Why is everything based on likelihoods even though likelihoods are so small?
#62Because it works well in practice. And to elaborate, usually when something works well in practice it's because it has multiple desirable properties - the one you "ask for", but also other ones you get for free. In this case maximum likelihood approximate bayesian estimation with a mostly reasonable prior. Furthermore you could look at the convergence properties which are good. You could probably design some degenera…
It's better than "it works well in practice". The question is misguided as stated. It's like asking why chemists care about density for measuring mass. If you are looking at the likelihood of any particular outcome of a continuous random variable, then you do not understand how probability works. The probability of any particular real number arising from a probability distribution on the real numbers is exactly 0. It…
Nitpicking somewhat, but e.g. `max(1, uniform(0, 2))` has a very non-zero probability of evaluating to 1.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#63Earlier quoted context omitted.
For continuous data it is exactly a probability density evaluated on your data (for discrete it's PMF instead). L(params)=P(D|params)
The point is that L(params) is not a probability density. The integral of L(params) over params is not one.
So we ARE working with a probability function. Its output can be interpreted as probabilities. It's just that we're maximizing L = P(events | params) with respect to params.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#64We're talking about the whole likelihood surface here, not just the single point that's the maximum likelihood estimator. The MLE is a method for choosing a valid point estimator from the likelihood function; it has some good properties, like being consistent (if you have enough data it converges to the truth) and asymptotically efficient (converges smallest possible variance) so long as some criteria are met.
But the MLE is not the only choice; for any given model, other procedures can be admissible estimators https://en.wikipedia.org/wiki/Admissible_decision_rule - it's just they also have to be procedures based on the likelihood function. In other words, your procedure doesn't have to be "take the likelihood function and find its maximum" but it has to be "take the likelihood function and... do something sensible with it."
So the MLE is popular in the frequentist world where you have to make the decision rules using the likelihood directly; in the Bayesian world, you take the likelihood and combine it with a prior, to make an actual probability distribution. Then you get things like like MAP (mode of the posterior) or the Bayes estimate (expectation of the posterior) - alternatives to MLE that still use the likelihood surface.
Of course this all works only if the underlying probabilistic model is literally true. So, in the machine learning world where the models are judged on being useful on usefulness and not expected to reflect mathematical reality, you're allowed to do things inconsistent with likelihood principle, like regularization tricks. In some physics situations (astronomical imaging comes to mind) where the probability model really is governed by the rules of nature, sticking to likelihood principle actually matters.
As to the question of being small, well, the likelihood is the probability (density) of the exact data you observe given parameters. Let's say you know the true parameter (the mean and standard deviation) and you observe a thousand draws from a normal distribution. Of course the probability of observing the very same pattern of a thousand values again is overwhelmingly unlikely. But if the mean was way different, that pattern would be proportionally even more unlikely. We should only care about relative probabilities. What's the probability that the universe evolved in exactly such a way that your cat will have exactly this fur pattern? Astronomically small. What's the probability that the universe evolved in such a way, and some of that fur ends up on your furniture? Another unimaginably small number. But what's the probability that, in a universe where you and your cat exist as you are, his fur will get everywhere? That's pretty much a certainty.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#65Earlier quoted context omitted.
For discrete distributions you indeed cannot, but for continuous distributions all you need is sufficiently small variance. Try for example a Gaussian with variance 1e-12
The value of a continuous probability density distribution at a specific point is pretty meaningless though; You have to talk about the integral between two values and that won't go above one.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#66Earlier quoted context omitted.
The point is that L(params) is not a probability density. The integral of L(params) over params is not one.
Right. But if you make the notation slightly more explicit, then the integral of L(data, params) over data is 1. This follows from the independence assumption. So we ARE working with a probability function. Its output can be interpreted as probabilities. It's just that we're maximizing L = P(events | params) with respect to params.
There is another function - a function of data for fixed params - which is a probability density. That doesn’t change the fact that the likelihood function isn’t.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#67Re: Why is everything based on likelihoods even though likelihoods are so small?
#68The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?” My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once. I already had a toolbox full of…
Re: Why is everything based on likelihoods even though likelihoods are so small?
#69In a continuous distribution the probability of any number on that distribution being generated is effectively zero. If R was generating the true probabilities it should give you zero for every single number. Think about it. That distribution is continuous over an infinite amount of numbers. If you select any number the chances of that number being generated will be essentially zero. According to the theory there is…
In a continuous distribution the probability of any number on that distribution being generated is effectively zero
Yes almost by definition, no ? You can only know the probability it would be between a range of numbers, by integrating over the distribution