Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why is everything based on likelihoods even though likelihoods are so small?

stats.stackexchange.com

Post reply on HN
Print view View original

81–90 of 103 posts

Re: Why is everything based on likelihoods even though likelihoods are so small?

#81

Post by kgwgk »

lupire wrote: post #80
TobyTheCamel wrote: post #49

Earlier quoted context omitted.

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the ma…

Concentration of mass is density . A shell is not dense. If I am looking for a needle in a hyperhaystack, it's not important to know that it's more likely to be "somewhere on the huge hyperboundary" than "in the center hypercubic inch".

And yet your parent comment was right in saying that it won't be true that "a lot of the probability mass - an amount that is not small - will be concentrated" in the center hypercubic inch.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#82

Post by shiandow »

For questions like these I sometimes prefer translating everything to thermodynamics.

In this case the question then becomes "Why are we looking at energy if at most ordinary temperatures systems aren't anywhere near 0 energy?". The overly simplified answer is "Because bowling balls still roll down, even at room temperature."

Which is actually kind of interesting to think about. Why do bowling balls roll down when they have vastly more thermal energy than gravitational potential? To get an answer you have to invoke the second law of thermodynamics. Which is annoyingly a bit hard to really get to understand. In short it implies that energy likes to become more disorganised, so something like gravitational potential which is very organised will eventually devolve into heat which is (by definition) disorganised. So bowling ball roll down.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#83

Post by knightoffaith »

cubefox wrote: post #76
knightoffaith wrote: post #57

Earlier quoted context omitted.

The actual probability is 0, but the probability density is not 0. Same reason why the probability that I pick 0.5 from a uniform distribution from 0 to 1 is 0, but the value of the probability density function of the distribution at 0.5 is 1.

What is this point value then measuring? A literal "density" doesn't seem plausible either, as points arguably do not have any "density".

I'll give the mathematical explanation. So if X is a continuous random variable, the probability that X takes on any particular value x is 0, i.e. P(X = x) = 0. However, it still makes sense to talk about P(X In the case of the cumulative distribution function of the uniform distribution from 0 to 1, since the derivative of x is 1, the probability distribution function is 1 from 0 to 1 and 0 elsewhere. This makes sense; the probability P(X < x) isn't increasing faster at one point than any other---with the exception of x outside of 0 and 1 having a probability density value of 0, since e.g. P(X < 2) is 100% and increasing the value of x=2 does not change this (it's still 100% because X only takes on values within [0,1]) .

Re: Why is everything based on likelihoods even though likelihoods are so small?

#84

Post by mturmon »

l_e_o_n wrote: post #34

You flip a possibly-biased coin 20 times and get half heads, half tails, e.g. "THHHTTTTTHTHTTHHHTHH". Under the model where the bias is 0.5—a fair coin—the probability of that sequence is (0.5)^20 or about one in a million. In fact, the probability of any sequence you could observe is one in a million. Under the model where the bias is 0.4 the probability is (0.4)^10 × (0.6)^10 or about one in two million. That is, t…

You have given a nice clean answer that does not make any errors (such as talking about the likelihood as a density in parameters, which of course it is not). Thanks for writing it down.

The only other thing worth adding to what you have written is that the likelihood is a product of N factors.

As such, it will essentially always diverge toward infinity (if the density factors are on average greater than 1) or collapse fast towards zero (if the factors are on average less than 1, as in your example and in OP).

So this very structure (arising from the IID observations) implies that no “stable” density will pop out. It’ll always blow up or down!

One way to stabilize things is to take (1/N) times the log of the likelihood. Then you will indeed converge to something familiar - the entropy, - E log p(x).

Re: Why is everything based on likelihoods even though likelihoods are so small?

#85

Post by NoPicklez »

hinkley wrote: post #58

The weirdest “likelihood” conversation I ever had, the putative team lead didn’t want to change priorities to fix a bug because, “how often does that happen?” My reply was, “it happens to every user, the first time they use the app.” And then something about how frequency has nothing to do with it. Every single user was going to encounter this bug. Even if they only used the app once. I already had a toolbox full of…

This is interesting, I wonder if there are principles from other types of engineering, civil, structural, aeronautical etc which provide some additional thoughts around probabilities of failure and now to deal with them

Particularly where you have a lot of low probability bugs

Re: Why is everything based on likelihoods even though likelihoods are so small?

#86

Post by devit »

Well, because any specific outcome from sampling a random distribution is indeed very unlikely.

In practice, that means that if you have an alternate "non-random" or "less random" explanation for the data, you'll be convinced that it's almost surely the correct one after just a few samples (via the obvious Bayesian decision framework, or just "common sense").

For example, imagine that you are rolling a die and 3 always comes up. On every roll, the likelihood of the die being a fair random die (as opposed to a loaded die) is divided by the number of sides, so with a 6-sided die, you'll usually be convinced that it's loaded after just 3-20 times of giving the same result (depending on your prior on it being loaded vs fair and your decision threshold).

Likewise, if only 1 and 2 come up you'll quickly be convinced that it's an unusual die that only has 1 and 2 symbols on the face.

Or another way to look at it is that processes are usually not random at all (rather they are usually deterministic, but the initial state is unknown) so a random distribution is a very bad model, and having any information about the initial state at all will drastically increase the likelihood and thus make the model with information strongly preferred; the likelihood of the random model is so low because that model is very bad, even though it may be the best available.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#87

Post by vermarish »

331c8c71 wrote: post #71
vermarish wrote: post #63

Earlier quoted context omitted.

Right. But if you make the notation slightly more explicit, then the integral of L(data, params) over data is 1. This follows from the independence assumption. So we ARE working with a probability function. Its output can be interpreted as probabilities. It's just that we're maximizing L = P(events | params) with respect to params.

The independence has nothing do with the integral being 1 to be honest. You could write a model where the observations are not independent but the (multivariate) integral over their domain will still be 1.

But for such a model, the joint pdf would not be written simply as a product of each individual pdf. That's what independence provides.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#88

Post by zmgsabst »

lupire wrote: post #80
TobyTheCamel wrote: post #49

Earlier quoted context omitted.

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This may be true for low dimensions but doesn’t generalise to high dimensions. Consider a 100-dimensional standard normal distribution for example. The MLE will still be at the origin but most of the ma…

Concentration of mass is density . A shell is not dense. If I am looking for a needle in a hyperhaystack, it's not important to know that it's more likely to be "somewhere on the huge hyperboundary" than "in the center hypercubic inch".

Disagree:

A lot of why large corporations fail to make products that people enjoy is tied up in this behavior and that mass is not independently distributed along each distribution — you end up with “continents of taste” your centroid product sucks for equally.

Re: Why is everything based on likelihoods even though likelihoods are so small?

#89

Post by LudwigNagasena »

agnosticmantis wrote: post #74
fergal_reid wrote: post #15

I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…

> However, the idea is that often a lot of the probability mass - an amount that is not small - will be concentrated around the maximum likelihood estimate, and so that's why it makes a good estimate, and worth using. This is a Bayesian point of view. The other answers are more frequentist, pointing out that likelihood at a parameter theta is NOT the probability of theta being the true parameter (given data). So we c…

That's not a Bayesian point of view. You can re-word it in terms of a confidence interval / coverage probability. It is true that in frequentist statistics parameters don't have probability distributions, but their estimators very much do. And one of the main properties of a good estimator is formulated in terms of convergence in probability to the true parameter value (consistency).

Re: Why is everything based on likelihoods even though likelihoods are so small?

#90

Post by LudwigNagasena »

kgwgk wrote: post #39
sega_sai wrote: post #4

It is very strange that this is on a main page. The key thing is likelihood is probability density of your data! I.e. if your probability density is a Gaussian N(0,0.00001), then the likelihoods of data-points next to the mean will be very large, if your PDF is N(0,10000) they'll be very small. Furthermore the amount of data matters as likelihoods will be multiplied for each datapoint, so if they were small in the be…

> The key thing is likelihood is probability density of your data! In fact the important thing to understand about the likelihood function is that it’s not a probability density.

I think there is some language misunderstanding going on. Likelihood function is not a probability density. Likelihood function evaluated for D is equal to probability density of D (by definition). In other words, f(x;theta) as a function of x is a probability density function. f(x;theta) as a function of theta is a likelihood function. But f(x;theta) for given x and theta is just a value, which, one can say, is both likelihood and probability density.
Post reply on HN