Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything based on likelihoods even though likelihoods are so small?

stats.stackexchange.com

Post reply on HN
Print view View original

41–50 of 103 posts

Re: Why is everything based on likelihoods even though likelihoods are so small?

#41

Post by l_e_o_n »

b10nic wrote: post #38
l_e_o_n wrote: post #34

You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…

Not quite; the probability of n/2 successes in n trials is given as Binomial(n,p) not p^n. p^n is correct for a single sequence but there are many possible sequences that result half heads, half tails and so you have a factor of "N choose X" or the so called "Binomial Coefficient". > (0.4)^20 × (0.6)^20 and I think you mean (0.4)^10 × (0.6)^10 or more generally p^x*(1-p)^n-x.

I'm talking about the whole sequence; you're talking about the number of heads (or) tails in the sequence.

The number of heads is a sufficient statistic, so we'll get the same likelihood ratios out, but the likelihood values themselves will be larger.

You could make a similar point about the original CrossValidated Normal(0, 1)^N example by summarizing the data with the mean and sum of squares.

This doesn't work if the data were Cauchy(0, 1)^N instead.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#42

Post by cj »

qup wrote: post #36
l_e_o_n wrote: post #34

You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…

> half heads, half tails. > Under the model where the bias is 0.5—a fair coin—the probability of that outcome is (0.5)^20 or about one in a million. No. Edit: someone downvoted, ha. It's closer to 1 in 6.

1 in a million is the probability of correctly predicting a unique sequence of 20 coin flips, in the exact order. (E.g. first 10 flips heads, 2nd 10 flips tails, in that order - 1 in a million)

Re: Why is everything based on likelihoods even though likelihoods are so small?

#43

Post by l_e_o_n »

qup wrote: post #36
l_e_o_n wrote: post #34

You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…

> half heads, half tails. > Under the model where the bias is 0.5—a fair coin—the probability of that outcome is (0.5)^20 or about one in a million. No. Edit: someone downvoted, ha. It's closer to 1 in 6.

Edited for clarity.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#44

Post by BeetleB »

qup wrote: post #36
l_e_o_n wrote: post #34

You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…

> half heads, half tails. > Under the model where the bias is 0.5—a fair coin—the probability of that outcome is (0.5)^20 or about one in a million. No. Edit: someone downvoted, ha. It's closer to 1 in 6.

I'm surprised people are conflating the Binomial distribution with OP's statement. He is talking about one specific outcome of half heads/half tails (where order matters). There is exactly one way to get that outcome.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#45

Post by sega_sai »

kgwgk wrote: post #39
sega_sai wrote: post #4

It is very strange that this is on a main page. The key thing is likelihood is probability density of your data! I.e. if your probability density is a Gaussian N(0,0.00001), then the likelihoods of data-points next to the mean will be very large, if your PDF is N(0,10000) they'll be very small. Furthermore the amount of data matters as likelihoods will be multiplied for each datapoint, so if they were small in the be…

> The key thing is likelihood is probability density of your data! In fact the important thing to understand about the likelihood function is that it’s not a probability density.

For continuous data it is exactly a probability density evaluated on your data (for discrete it's PMF instead).

L(params)=P(D|params)

Re: Why is everything based on likelihoods even though likelihoods are so small?

#46

Post by fergal_reid »

crazygringo wrote: post #37
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> Yes, individual likelihoods are so small, that yes even a MLE solution is extremely unlikely to be correct. Can you elaborate? An MLE is never going to come up with the exact parameters that produced the samples, but in the original example, as long as you know it's a normal distribution, MLE is probably going to come up with a mean between 4 and 6 and a SD within a similar range as well (I haven't calculated it, j…

Right - I think this is what's at the heart of the original question.

I know they asked with a continuous example, but I don't interpret their question as limited to continuous cases, and I think it's easier to address using a discrete example, as we avoid the issue of each exact parameter having infinitesimal mass which occurs in a continuous setting.

Let's imagine the parameter we're trying to estimate is discrete and has, say, 500 different possible values.

Let's say the parameter can have the value of the integers between 1 and 500 and most of the mass is clustered in the middle between 230 and 270.

Given some data, it would actually be possible that MLE would come up with the exact value, say 250.

But maybe given the data, a range of values between 240 and 260 are also very plausible, so the likelihood of exactly 250 has a fairly low probability.

The original poster is confused, because they are basically saying, well, if the actual probability is so low, why is this MLE stuff useful?

You are pointing out they should really frame things in terms of a range and not a point estimate. You are right; but I think their question is still legitimate, because often in practice we do not give a range, and just give the maximum likelihood estimate of the parameter. (And also, separately, in a discrete parameter setting, specific parameter value could have substantial mass.)

So why is the MLE useful?

My answer would be, well, that's because for many posterior distributions, a lot of the probability mass will be near the MLE, if not exactly at it - so knowing the MLE is often useful, even if the probability of that exact value of the parameter is low.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#47

Post by kgwgk »

sega_sai wrote: post #45
kgwgk wrote: post #39

Earlier quoted context omitted.

> The key thing is likelihood is probability density of your data! In fact the important thing to understand about the likelihood function is that it’s not a probability density.

For continuous data it is exactly a probability density evaluated on your data (for discrete it's PMF instead). L(params)=P(D|params)

The point is that L(params) is not a probability density. The integral of L(params) over params is not one.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#49

Post by TobyTheCamel »

fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using.

This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the mass will live in a thin shell of distance roughly 7 units from the origin.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#50

Post by knightoffaith »

bscphil wrote: post #30
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> It is fair to ask why the likelihoods are useful if they are so small The way the question demonstrates "smallness" is wrong, however. They quote the product of the likelihoods of 50 randomly sampled values - 9.183016e-65 - as if the smallness of this value is significant or meant anything at all. Forget the issue of continuous sampling from a normal distribution, and just consider the simple discrete case of flipp…

Right - and so the more appropriate thing to do is not look at the raw likelihood of any one particular value but instead look at relative likelihoods to understand what values are more likely than other values.
Post reply on HN