Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything based on likelihoods even though likelihoods are so small?

stats.stackexchange.com

Post reply on HN
Print view View original

71–80 of 103 posts

Re: Why is everything based on likelihoods even though likelihoods are so small?

#71

Post by 331c8c71 »

vermarish wrote: post #63
kgwgk wrote: post #47

Earlier quoted context omitted.

The point is that L(params) is not a probability density. The integral of L(params) over params is not one.

Right. But if you make the notation slightly more explicit, then the integral of L(data, params) over data is 1. This follows from the independence assumption. So we ARE working with a probability function. Its output can be interpreted as probabilities. It's just that we're maximizing L = P(events | params) with respect to params.

The independence has nothing do with the integral being 1 to be honest. You could write a model where the observations are not independent but the (multivariate) integral over their domain will still be 1.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#72

Post by 331c8c71 »

kgwgk wrote: post #47
sega_sai wrote: post #45

Earlier quoted context omitted.

For continuous data it is exactly a probability density evaluated on your data (for discrete it's PMF instead). L(params)=P(D|params)

The point is that L(params) is not a probability density. The integral of L(params) over params is not one.

That's exactly the reason why frequentist approach sucks by the way;) Parameters are treated specially and there is no internal consistency - to have it you need to introduce priors...

Re: Why is everything based on likelihoods even though likelihoods are so small?

#73

Post by 331c8c71 »

ninetyninenine wrote: post #22

In a continuous distribution the probability of any number on that distribution being generated is effectively zero. If R was generating the true probabilities it should give you zero for every single number. Think about it. That distribution is continuous over an infinite amount of numbers. If you select any number the chances of that number being generated will be essentially zero. According to the theory there is…

> If R was accurately calculating likelihood it should give you zero for each number.

You have some good points but this is false. The probability of any point for a continuous distribution is indeed zero. That doesn't mean that the density at this point is also zero.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#74

Post by agnosticmantis »

fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using.

This is a Bayesian point of view. The other answers are more frequentist, pointing out that likelihood at a parameter theta is NOT the probability of theta being the true parameter (given data). So we can't and don't interpret it like a probability.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#75

Post by klipt »

agnosticmantis wrote: post #74
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This is a Bayesian point of view. The other answers are more frequentist, pointing out that likelihood at a parameter theta is NOT the probability of theta being the true parameter (given data). So we c…

Given enough data, Bayesian and frequentist models tend to converge to the same answer anyway.

Bayesian priors have similar effect to regularization (e.g. ridge regression / penalizing large parameter values).

Re: Why is everything based on likelihoods even though likelihoods are so small?

#76

Post by cubefox »

knightoffaith wrote: post #57
mathisd wrote: post #55

I don't understand why maximum of likelyhood is not zero in the example given. Isn't P(X = x / theta = theta_0) always null for continuous laws ?

The actual probability is 0, but the probability density is not 0. Same reason why the probability that I pick 0.5 from a uniform distribution from 0 to 1 is 0, but the value of the probability density function of the distribution at 0.5 is 1.

What is this point value then measuring? A literal "density" doesn't seem plausible either, as points arguably do not have any "density".

Re: Why is everything based on likelihoods even though likelihoods are so small?

#77

Post by anon946 »

bscphil wrote: post #30
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> It is fair to ask why the likelihoods are useful if they are so small The way the question demonstrates "smallness" is wrong, however. They quote the product of the likelihoods of 50 randomly sampled values - 9.183016e-65 - as if the smallness of this value is significant or meant anything at all. Forget the issue of continuous sampling from a normal distribution, and just consider the simple discrete case of flipp…

For the discrete case, it seems that a better thing to do is consider the likelihood of getting that number of heads, rather than the likelihood of getting that exact sequence.

I am not sure how to handle the continuous case, however.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#78

Post by 3abiton »

331c8c71 wrote: post #72
kgwgk wrote: post #47

Earlier quoted context omitted.

The point is that L(params) is not a probability density. The integral of L(params) over params is not one.

That's exactly the reason why frequentist approach sucks by the way;) Parameters are treated specially and there is no internal consistency - to have it you need to introduce priors...

It's the bayes vs frequentist war again.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#79

Post by lupire »

anon946 wrote: post #77
bscphil wrote: post #30

Earlier quoted context omitted.

> It is fair to ask why the likelihoods are useful if they are so small The way the question demonstrates "smallness" is wrong, however. They quote the product of the likelihoods of 50 randomly sampled values - 9.183016e-65 - as if the smallness of this value is significant or meant anything at all. Forget the issue of continuous sampling from a normal distribution, and just consider the simple discrete case of flipp…

For the discrete case, it seems that a better thing to do is consider the likelihood of getting that number of heads, rather than the likelihood of getting that exact sequence. I am not sure how to handle the continuous case, however.

Of course you ignore irrelevant ordering of data points. That's not the issue.

The issue, for discrete or continuous (which are mathematically approximations of each other), is that the value at a point is less important than the integral over a range. That's why standard deviation is useful. The argmax is a convenient average over a weightable range of values. The larger your range, the greater the likelihood that the "truth" is in that range.

If you only need to be correct up to 1% tolerance, the likelihood of a range of values that have $SAMPLING_PRECISION tolerance is not importance. Only the argmax is, to give you a center of the range.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#80

Post by lupire »

TobyTheCamel wrote: post #49
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the ma…

Concentration of mass is density. A shell is not dense.

If I am looking for a needle in a hyperhaystack, it's not important to know that it's more likely to be "somewhere on the huge hyperboundary" than "in the center hypercubic inch".

Post reply on HN