Relative likehoods can be used for prioritization.
Why is everything based on likelihoods even though likelihoods are so small?
31–40 of 103 posts
Re: Why is everything based on likelihoods even though likelihoods are so small?
#32I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…
I agree with your points and thats why it's useful to compare a MLE to an alternative model via a likelihood ratio test, in which case one sees how much better the generative model performs as compared to the wrong model.
Similarly, AIC values do not make a lot of sense on an absolute scale but only relative to each other, as written in [1].
[1] Burnham, K. P., & Anderson, D. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological methods & research, 33(2), 261-304.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#33One might perhaps say that what is probable
Is that men experience much that is improbable.
- Fragment of a lost play by Agathon, ca. 2,400 years ago, quoted in Aristotle's The Art of Rhetoric
Re: Why is everything based on likelihoods even though likelihoods are so small?
#34You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH".
Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million.
Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million.
That is, the sequence we observed supplies about twice as much evidence in favor of bias = 0.5 as compared with bias = 0.4—this is likelihood.
Likelihood ratios are all that matter.
Morals:
- The more complex the event you're predicting (the rarer the tyical observed result) the smaller the associated likelihoods will tend to be
- It's possible that every observed result has a tiny probability under every model you're considering
- Nonetheless it makes sense to use the ratios of these numbers to compare the models
- This has nothing to do with probability densities or logarithms, though the fact that we often work with densities also makes absolute likelihood values relative to the choice of units
Added in edits:
- You could summarize the sequence with the number of heads or tails and then the likelihood values would be larger but the ratios would remain the same (it's a sufficient statistic). Similarly in the CrossValidated question one could summarize the data with the mean and sum of squares. But this doesn't work in general, e.g. if we have i.i.d. draws from a Cauchy distribution.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#35Earlier quoted context omitted.
Wait how do you get likelihoods greater than one? Definitely won’t work for likelihoods like Poisson or other count based models.
For discrete distributions you indeed cannot, but for continuous distributions all you need is sufficiently small variance. Try for example a Gaussian with variance 1e-12
The value of a continuous probability density distribution at a specific point is pretty meaningless though; You have to talk about the integral between two values and that won't go above one.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#36You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…
> half heads, half tails.
> Under the model where the bias is 0.5—a fair coin—the probability of that outcome is (0.5)^20 or about one in a million.
No.
Edit: someone downvoted, ha. It's closer to 1 in 6.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#37I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…
> Yes, individual likelihoods are so small, that yes even a MLE solution is extremely unlikely to be correct.
Can you elaborate? An MLE is never going to come up with the exact parameters that produced the samples, but in the original example, as long as you know it's a normal distribution, MLE is probably going to come up with a mean between 4 and 6 and a SD within a similar range as well (I haven't calculated it, just eyeballing it) -- when the original parameters were 5 and 5.
I guess I don't know what you mean by "correct", but that's as correct as you can get, based on just 50 samples.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#38You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…
Not quite; the probability of n/2 successes in n trials is given as Binomial(n,p) not p^n. p^n is correct for a single sequence but there are many possible sequences that result half heads, half tails and so you have a factor of "N choose X" or the so called "Binomial Coefficient".
> (0.4)^20 × (0.6)^20
and I think you mean (0.4)^10 × (0.6)^10 or more generally p^x*(1-p)^n-x.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#39It is very strange that this is on a main page. The key thing is likelihood is probability density of your data! I.e. if your probability density is a Gaussian N(0,0.00001), then the likelihoods of data-points next to the mean will be very large, if your PDF is N(0,10000) they'll be very small. Furthermore the amount of data matters as likelihoods will be multiplied for each datapoint, so if they were small in the be…
> The key thing is likelihood is probability density of your data!
In fact the important thing to understand about the likelihood function is that it’s not a probability density.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#40> Thus, it appears to be very unlikely in a certain sense that these numbers came from the very distribution they were generated from.
“It appears to be very unlikely in a certain sense that this comment is written in English. Yes, this sequence of characters is much more likely in English than in French. But you can’t even fathom how unlikely it was to be ever written in English!”