Why is everything based on likelihoods even though likelihoods are so small?

Some bugs have multiple symptoms, it’s true.

If by “joint probability” you mean function(params, data) there is no joint probability here in general.

L(params, data) is constructed from a family density functions p(data) for each possible value of param. The integral of L(params, data) may be anything or diverge. You don’t need any extra independence assumption either.

Or maybe you mean “joint probability” as p(data1, data2) when data is composed of two observations, for example. But you don’t need any independence assumption for that probability density to integrate to one! It necessarily does that - whether you can factorize it as p’(data1)p’’(data2) or not.

Everyone seems to be missing the point here. The SO post says:

> As we can see, even from the correct distribution, the likelihood is very, very small. Thus, it appears to be very unlikely in a certain sense that these numbers came from the very distribution they were generated from.

The person who asked the question is simply confused between likelihoods and posterior probabilities. The likelihood of d values from a Normal Distribution is defined to be the probability of sampling those d values given the parameters of the Normal. It is not the probability that those numbers came from that Normal Distribution. To answer the latter question, you need to say what other possibilities you're considering (perhaps some other parameter values) and use Bayes Rule. The other answers mention that ratios will be involved, but the way to see why ratios are involved is to look at Bayes Rule.

That's interesting and intuitive for a uniform distribution. What does it then mean on a non-uniform distribution for an value to be very small? Is there some interpretation for that? The Stack Overflow post actually mentions values that are extremely close to zero.

So, just to be sure, even for a uniform distribution, the values can be small. Consider the uniform distribution from 0 to 10^100. The CDF for this distribution is P(X So PDFs can have small values regardless of whether they are uniform or not. What a small PDF at a point x indicates is that the CDF is increasing very "slowly" at that x. I'll emphasize this point - PDF values are not probabilities. They are rates of change of the CDF.

For some further understanding of the stack overflow post, let's consider Uniform(0, 2). The PDF is p(x) = 1/2. Suppose the author of the stack overflow post drew 50 samples from this distribution. Regardless of what those 50 samples were, the value he would have gotten would have been (1/2)^50 = 1/(2^50), something on the order of 10^-16. Why is this so small?

(I'll give a rather loose and informal explanation here, but I can be more formal if you'd like, if this doesn't make sense.) Think back to Uniform(0, 1) vs. Uniform(0, 10^100). Recall that the probability that a particular x falls in [0, 1] for the former distribution is the same as the probability that a particular x falls in [0, 10^100]---i.e. 1 (100%). In the case of the latter distribution, that 1 has had to be "spread out" across a larger space, which should give some intuition as to why the PDF is low---for a particular unit in space that we "travel", since the probability has been spread out so thinly across the space, the CDF isn't increasing that much, i.e. the PDF isn't that high.

When we're looking at PDF values when we're looking at the space of possibilities covered by 50 samples, it's going to be a lot "larger" than the space covered by 1 sample (over one sample, the space is [0,2], covering 2 units of space. over two samples, the space is the square [0,2] x [0,2], with an area of 4. over 50 samples, the space is the hypercube [0,2]^50, with a 50-dimensional volume of 2^50---a huge space.) But the total probability is still 1, so it's going to be "spread out" very thinly across this larger space, hence much smaller values. And so, the probability we accumulate as we move across this space per unit is going to be very low, hence a low likelihood value.

So when we draw many samples from a distribution, the likelihood of these samples is going to be very small (mostly---there might be spikes where they're high).

I've spoken a little loosely and informally, but hopefully this makes sense.