You work with probability density functions because the probability of observing any given value in a continuum is zero. Density functions may be reasonable to work with if they have some nice properties (continuity, unimodality, ...) The question and answers here seem to be from people that don't understand calculus.
Why is everything based on likelihoods even though likelihoods are so small?
21–30 of 103 posts
Re: Why is everything based on likelihoods even though likelihoods are so small?
#22In a continuous distribution the probability of any number on that distribution being generated is effectively zero. If R was generating the true probabilities it should give you zero for every single number.
Think about it. That distribution is continuous over an infinite amount of numbers. If you select any number the chances of that number being generated will be essentially zero. According to the theory there is no possibility for any number on the distribution to be generated. This is correct.
Yet when you use the random number generator you get a number even though that number technically is impossible to exist due to zero probability. Does this mean there is a flaw in the theory when applied to the number generated?
Yes it does. The theory is an approximation of what's going on itself. No random number generator in a computer is selecting a number from a truly continuous set of numbers. It is selecting it from a finite set of numbers from all available numbers in a floating point specification.
Even if it's not a computer when you select a random number by intuition from a continuous distribution you are not doing it randomly.
Think about it. Pick a random number between 0 and 1. I pick 0.343445434. This selection is far from random. It is biased because there is an infinite amount of significant figures yet I arbitrarily don't go past a certain amount. I cut off at 9 sigfigs and bias towards a cutoff like that because picking a random number with say 6000 sigfigs is just too inconvenient. You really need to account for infinite sigfigs for the number to be truly random which is impossible.
So even when you pick numbers randomly you are actually picking from a finite set.
In fact I can't think of anything in reality that can truly be accurately described with a continuous distribution. Nothing is in the end truly continuous. Or maybe it does exist, but if it does exist how can we even confirm it? We can't verify anything in reality to a level of infinite sig figs.
If R was accurately calculating likelihood it should give you zero for each number. And the random number generator should not even be able to exist as how do even create a pool of infinite possibilities to select from? Likely R is giving some probability over a small interval of numbers.
That's where the practicality of the continuous distribution makes sense when you measure the probability of a range of values. You get a solid number in this case.
Anyway the above explanation is probably too deep. A more practical way of thinking about this is like this:
It is unlikely for any one person to win the lottery. Yet someone always wins. The probability of someone winning is 100 percent. The probability of a specific someone winning is 1 over the total number of people playing.
Improbable events in the universe happen all the time because that all that's available. It's highly improbable for any one person to win the lottery but if someone has to win, then there is a 100 percent chance that an arbitrary improbable event will occur.
This is more easily seen in a uniform discrete distribution rather then the normal continuous distribution.
In the case of the normal distribution it is confusing. In a normal distribution It is far more likely for an improbable event to occur then it is for the single most probable event to occur.
Think of it like this. I have a raffle. There are 2 billion participants. Each person has one ticket in the bag, except me. I have 100,000 tickets in the bag I am the most likely person to win.
But it is still far more likely for anyone else but me to win even when I am the most likely person to win. An arbitrary improbable event is more likely to occur then the single most probable event.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#23For infinite probability spaces, likelihood has no interpretable meaning akin to probability in the case of finite spaces. This can be a source of confusion.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#24In a storm, the likelihood of A raindrop hitting your computer dwarfs the likelihood of THE raindrops[5927] hitting your computer.
Hence why using A and THE is important.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#25Another thing to note is that you're multiplying probabilities together. Since each probability is between 0 and 1, youre always shrinking the likelihood with each new data point. When you're doing this kind of analysis, the question you're asking is "given a model with these parameters, what's the probability I get exactly this sample?" Which, when you phrase it that way, it becomes more apparent why the likelihood…
Multiplying them together certainly magnifies the effect, but it would magnify it the other way if the likelihoods were larger than one. (Easy to get, just tweak the variance of the normal distributions to be smaller). Likelihoods are more like infinitesimal fractions of a probability, that need to be integrated over some set of events to get back a probability. In the case of the joint distribution of 50 Gaussian, y…
Wait how do you get likelihoods greater than one?
Definitely won’t work for likelihoods like Poisson or other count based models.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#26I think the correct answer is that it is mostly bogus, but likelihood based statistical methods mostly work for well-behaving distributions, especially for Gaussian.
Maximum likelihood estimation has some weird cases when the distribution is not "well behaving".
Re: Why is everything based on likelihoods even though likelihoods are so small?
#27Probability is the probability mass distributed over your data with fixed parameters, and likelihood is mass distributed over your model parameters with fixed data. The absolute most important thing to know about likelihood is that it is not a measure of probability, even thought it looks a lot like probability.
If I look at coin flip data, I know the data comes from a coin flip, but any specific count of heads vs tails becomes less and less likely the more flips we do. So likelihood being small tells us nothing on its own.
The value of likelihood comes from the framework you use in. If I wanted to make a best guess at what the balance of the coin is then I could find the maximum of the likelihood over all coin balances to get the most representative version of my model. Similarly, I can compare two specific coin biases and determine which is more likely, but that alone can’t tell me anything about the probability of the coin being biased.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#28Earlier quoted context omitted.
Multiplying them together certainly magnifies the effect, but it would magnify it the other way if the likelihoods were larger than one. (Easy to get, just tweak the variance of the normal distributions to be smaller). Likelihoods are more like infinitesimal fractions of a probability, that need to be integrated over some set of events to get back a probability. In the case of the joint distribution of 50 Gaussian, y…
Wait how do you get likelihoods greater than one? Definitely won’t work for likelihoods like Poisson or other count based models.
For discrete distributions you indeed cannot, but for continuous distributions all you need is sufficiently small variance. Try for example a Gaussian with variance 1e-12
Re: Why is everything based on likelihoods even though likelihoods are so small?
#29I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…
Yes - the most enlightening concept for me was "Highest Probability Density Interval" which basically always is clustered around the mean. But you can choose any interval which contains as much probability mass!
https://en.wikipedia.org/wiki/Credible_interval#Choosing_a_c...
It's a fairly common "mistake" to assume that the MLE is useful as a point estimate and without considering covariance/spread/CI/HPDI/FIM/CRLB/Entropy/MI/KLD or some other measure of precision given the measurement set.
Re: Why is everything based on likelihoods even though likelihoods are so small?
#30I think most of the replies, here and on stack exchange, are answering slightly the wrong question. It is fair to ask why the likelihoods are useful if they are so small, and it's not a good answer to talk about how they could be expressed as logs, or even to talk about the properties of continuous distributions. I think the answer is: Yes, individual likelihoods are so small, that yes even a MLE solution is extremel…
> It is fair to ask why the likelihoods are useful if they are so small
The way the question demonstrates "smallness" is wrong, however. They quote the product of the likelihoods of 50 randomly sampled values - 9.183016e-65 - as if the smallness of this value is significant or meant anything at all. Forget the issue of continuous sampling from a normal distribution, and just consider the simple discrete case of flipping a coin. The combined probability of any permutation of 50 flips is 0.5 ^ 50, a really small number. That's because the probability is, in fact, really small!