[dead]
Why is everything based on likelihoods even though likelihoods are so small?
11–20 of 103 posts
Re: Why is everything based on likelihoods even though likelihoods are so small?
#12Because it works well in practice. And to elaborate, usually when something works well in practice it's because it has multiple desirable properties - the one you "ask for", but also other ones you get for free. In this case maximum likelihood approximate bayesian estimation with a mostly reasonable prior. Furthermore you could look at the convergence properties which are good. You could probably design some degenera…
It's better than "it works well in practice".
The question is misguided as stated. It's like asking why chemists care about density for measuring mass.
If you are looking at the likelihood of any particular outcome of a continuous random variable, then you do not understand how probability works.
The probability of any particular real number arising from a probability distribution on the real numbers is exactly 0. It's not an arbitrarily small epsilon greater than zero, it's actually zero. This definition is in fact required for probability to sense mathematically.
You might ask questions like why does maximum likelihood work as an optimization criterion, but that's very different from asking why we care about likelihood at all.
The comments on the original question do a good job of cutting through this confusion.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#13It is very strange that this is on a main page. The key thing is likelihood is probability density of your data! I.e. if your probability density is a Gaussian N(0,0.00001), then the likelihoods of data-points next to the mean will be very large, if your PDF is N(0,10000) they'll be very small. Furthermore the amount of data matters as likelihoods will be multiplied for each datapoint, so if they were small in the be…
I'm surprised it's here as well. Of all the interesting questions on CV, I would not consider this one of them. I wonder if this was sent through the second-chance pool.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#14Another thing to note is that you're multiplying probabilities together. Since each probability is between 0 and 1, youre always shrinking the likelihood with each new data point. When you're doing this kind of analysis, the question you're asking is "given a model with these parameters, what's the probability I get exactly this sample?" Which, when you phrase it that way, it becomes more apparent why the likelihood…
Multiplying them together certainly magnifies the effect, but it would magnify it the other way if the likelihoods were larger than one. (Easy to get, just tweak the variance of the normal distributions to be smaller). Likelihoods are more like infinitesimal fractions of a probability, that need to be integrated over some set of events to get back a probability. In the case of the joint distribution of 50 Gaussian, you can think of the likelihood having "units" of epsilon^50.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#15I think most of the replies, here and on stack exchange, are answering slightly the wrong question.
It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions.
I think the answer is:
Yes, individual likelihoods are so small, that yes even a MLE solution is extremely unlikely to be correct.
However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using.
Much like how the average is unlikely to be the exact value of a new sample from the distribution, but it's a good way of describing what to expect. (And gets better if you augment it with some measure of dispersion, and so on). (If the distribution is very dispersed, then while the average is less useful as an idea of what to expect, it still minimises prediction error in some loss; but that's a different thing and I think less relevant here).
Re: Why is everything based on likelihoods even though likelihoods are so small?
#16I don't have a problem with very small probabilities as long as they stay within math and kind of "cancel out".
What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events.
For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?
Re: Why is everything based on likelihoods even though likelihoods are so small?
#17I don't have a problem with very small probabilities as long as they stay within math and kind of "cancel out". What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events. For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?
Quite a lot. See airbags, seat belts, etc.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#18I don't have a problem with very small probabilities as long as they stay within math and kind of "cancel out". What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events. For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?
One simple way of quantifying this is (amt/cost of harm) * (risk of occurrence)
Re: Why is everything based on likelihoods even though likelihoods are so small?
#19I don't have a problem with very small probabilities as long as they stay within math and kind of "cancel out". What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events. For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?
> For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?
if you value being killed at a massively negative value, then 1/10,000 times that value is still a massively negative value, so the answer is "a huge amount of effort"
Re: Why is everything based on likelihoods even though likelihoods are so small?
#20How I'd put it:
They are useful because the integral of the likelihoods is not infinitesimal.
The probability that your yard stick measures 1.0000000000000 yards is basically zero, but the probability that it's within one inch of that is close to one.
We generally prefer to use probability density functions with the property that most of the probability density is close to the maximum likelihood.
So, in the yard stick example, the yard stick lengths are probably gaussian, so if check enough lengths, you'll get a mean (== length with the maximum likelihood) that approaches 1.00000000000 yards (you'd hope) with some small standard deviation (probably less than an inch).