Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything based on likelihoods even though likelihoods are so small?

stats.stackexchange.com

Post reply on HN
Print view View original

51–60 of 103 posts

Re: Why is everything based on likelihoods even though likelihoods are so small?

#51

Post by blt »

TobyTheCamel wrote: post #49
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the ma…

I think the "mass" they are referring to might the mass of the Bayesian posterior in parameter space, not the mass of the data in event space.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#52

Post by fergal_reid »

blt wrote: post #51
TobyTheCamel wrote: post #49

Earlier quoted context omitted.

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the ma…

I think the "mass" they are referring to might the mass of the Bayesian posterior in parameter space, not the mass of the data in event space.

Yes, in parameter space.

However, TobyTheCamel's point is valid in that there are some parameter spaces where the MLE is going to be much less useful than others.

Even without having to go to high dimensions, if you've got a posterior that looks like a normal distribution, the MLE is going to the you a lot, whereas if it's a multimodal distribution with a lot of mass scattered around, knowing the MLE much less informative.

But this is a complex topic to address in general, so I'm trying to stick to what I see as the intuition behind the original question!

Re: Why is everything based on likelihoods even though likelihoods are so small?

#53

Post by samatman »

airstrike wrote: post #19
praptak wrote: post #16

I don't have a problem with very small probabilities as long as they stay within math and kind of "cancel out". What I do have a problem with is lack of conceptual framework for dealing with small probabilities of real life events. For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times?

> For example, what amount of effort is appropriate to prevent a one time event which kills you with say 1 in ten thousand times? if you value being killed at a massively negative value, then 1/10,000 times that value is still a massively negative value, so the answer is "a huge amount of effort"

For some value of "massive", sure. But for any value of massive, it's 1/10,000th that value. Then you factor in the value derived from taking that risk, and there's your choice.

The reality is that I don't expend huge amounts of effort avoiding tail risks, and you don't either. You might for the ones you're explicitly aware of, but a risk is a risk whether or not you know you're taking it.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#54

Post by lukego »

Likelihoods aren’t fundamentally small.

The center of a normal distribution has high likelihood (e.g. 1000000) if the standard deviation is small or low likelihood if the standard deviation is large (e.g. 1/1000000.)

This effect is amplified when you are working with products of likelihoods. They can be infinitesimal or astronomical.

Giant likelihoods really surprised me the first time I experienced them but they’re not uncommon when you work with synthetic test data in high dimensions and/or small scales.

They still integrate to the same magnitude because the higher likelihood values are spread over shorter spans.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#56

Post by kjhcvkek77 »

nerdponx wrote: post #12
kjhcvkek77 wrote: post #3

Because it works well in practice. And to elaborate, usually when something works well in practice it's because it has multiple desirable properties - the one you "ask for", but also other ones you get for free. In this case maximum likelihood approximate bayesian estimation with a mostly reasonable prior. Furthermore you could look at the convergence properties which are good. You could probably design some degenera…

It's better than "it works well in practice". The question is misguided as stated. It's like asking why chemists care about density for measuring mass. If you are looking at the likelihood of any particular outcome of a continuous random variable, then you do not understand how probability works. The probability of any particular real number arising from a probability distribution on the real numbers is exactly 0. It…

I appreciate your response but I don't really agree. They say that likelihood can be multiplied by any scale factor or that it's only the comparative difference that matters, or we can make a little plot, but they don't actually explain why.

I can try to make an explanation from the bayesian framework(but as I mentioned it's not the only relevant one)

Likelihood is P(measurement=measurement'|parameter=parameter'). This is a small value. Given a prior we can P(parameter=parameter'|measurement=measurement'). This is also small. But when we compute P(parameter'-kP(X_i|Y) = (P(X_i)P(Y|X_i)/(sum_j P(X_j)P(Y|X_j))

I'm obviously skipping a lot of steps here because I'm sketching an explanation rather than giving one.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#57

Post by knightoffaith »

mathisd wrote: post #55

I don't understand why maximum of likelyhood is not zero in the example given. Isn't P(X = x / theta = theta_0) always null for continuous laws ?

The actual probability is 0, but the probability density is not 0. Same reason why the probability that I pick 0.5 from a uniform distribution from 0 to 1 is 0, but the value of the probability density function of the distribution at 0.5 is 1.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#58

Post by hinkley »

The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?”

My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once.

I already had a toolbox full of conversations about how bad we are at statistics, but that one opened up a whole new avenue of things to worry about. One that was reinforced by later articles about the uselessness of p95 stats - particularly where 3% of your users are experiencing 100% outage.

But the one that is more apropos to the linked question, vs HN in general, is how people are bad at calculating the probability that “nothing bad happens” when there are fifty low probability things that can go wrong. Especially as the number of opportunities go up.

And the way that, if we do something risky and nothing bad happens, we estimate down the probability of future calamity instead of counting ourselves lucky and backing away.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#59

Post by fwungy »

Yes. Statistics are fragile indicators well beyond the Central Limit Theorem minimal sufficient boundaries. They work pretty good when you have tons of data and run tons of repetitions, but for moderate sized data and repetitions you need very high certainty levels for statistics to help much.

You can play perfect blackjack and card count at a table with good rules and lose plenty because your advantage is small (Statistics get even worse when the probabilities are chained because the weakest estimator bounds the rest.

Essentially, if you always follow statical advice you should do better than average, if you're lucky. There are better heuristics than statistics in most fields of human decision making.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#60

Post by opportune »

hinkley wrote: post #58

The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?” My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once. I already had a toolbox full of…

I've seen this exact same fallacy happen several times throughout my career, which isn't even very long.

I think in many cases it boils down to some subtype not being identified and evaluated on its own. As in your case it's especially impactful, and yet IME also usually where these kinds of things get improperly prioritized, when it's a user's first impression or when it occurs in a way that causes a user to have to just sit and wait on the other end as these are often "special" cases with different logic in your application code.

OTOH sometimes users try to weird/wrong/adversarial shit and so their high failure rate is working as intended. But it pollutes your stats such that it can hide real issues with similar symptoms and skew distributions.

Post reply on HN