Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Maximum likelihood estimation and loss functions

rish-01.github.io

Post reply on HN
Print view View original

21–30 of 33 posts

Re: Maximum likelihood estimation and loss functions

#21

Post by alalv »

dumb1224 wrote: post #20
levocardia wrote: post #7

Somehow I've never found an explanation for MLE that's intuitive and suitable for someone who didn't take graduate-level statistics already. I'm 100% on board with the introduction (MSE and cross-entropy make total intuitive sense; you can see how they penalize 'wrongness' to an increasing degree) but in the next paragraph we jump right to: >Let pmodel(x;θ) be a parametric family of distributions over a space of para…

I was doing machine learning but never dig into stats before. Then I tried to study Bayesian inference and regression by myself and finally I got what it really means and its importance. First I realised that ubiquity of 'likelihood' and 'likelihood function', then I realised it's just a way to parameterise the model parameters instead of input data. Then MLE is a way to get an estimate of maximum of that function, w…

It is not the most likely setting to give rise to the data observed (that is the posterior), is the setting in which the data observed is the most likely.

Re: Maximum likelihood estimation and loss functions

#22

Post by truculent »

anon946 wrote: post #19

My favorite MLE example: Suppose you walk into a bank and ask them to give you a quarter. You flip the quarter twice and get two heads. Given this experiment, what do you estimate to be the probability p of getting a heads when you flip this coin? Using MLE, you would get p = 1. In other words, this coin will always give you a heads when you flip it! (According to MLE.)

The example only seems ridiculous because you've deliberately excluded relevant knowledge about the world from the model. Add a prior to the model and you'll have a much more reasonable function to maximise.

Re: Maximum likelihood estimation and loss functions

#23

Post by unnah »

Myrmornis wrote: post #10
levocardia wrote: post #7

Somehow I've never found an explanation for MLE that's intuitive and suitable for someone who didn't take graduate-level statistics already. I'm 100% on board with the introduction (MSE and cross-entropy make total intuitive sense; you can see how they penalize 'wrongness' to an increasing degree) but in the next paragraph we jump right to: >Let pmodel(x;θ) be a parametric family of distributions over a space of para…

One initial thing to understand is that the probability mass/density functions that you get taught in connection with standard probability distributions (binomial, Normal, etc) are functions of the data values : you put in a data value and the function outputs a probability (density), for some fixed parameter values. At first glance likelihood functions might look the same, but you have to think of them as functions…

An important point here is that the integral of the likelihood function over different parameter values is not constrained to be 1. This is why a likelihood is not a probability or a probability density, but its own thing. The confusing bit is that the likelihood formula is exactly the same as the formula of the original probability density function...

Re: Maximum likelihood estimation and loss functions

#24

Post by PunchTornado »

jey wrote: post #8
levocardia wrote: post #7

Somehow I've never found an explanation for MLE that's intuitive and suitable for someone who didn't take graduate-level statistics already. I'm 100% on board with the introduction (MSE and cross-entropy make total intuitive sense; you can see how they penalize 'wrongness' to an increasing degree) but in the next paragraph we jump right to: >Let pmodel(x;θ) be a parametric family of distributions over a space of para…

This notation doesn't require graduate-level statistics knowledge, it's more like stuff that would be covered in a first mathematical course on probability and statistics. It's totally practical to learn this stuff on your own from videos, books, and PDFs. First you need to get a solid conceptual grasp on probability distributions and their associated concepts like random variables, conditional probability, and joint…

I took stats courses and didn’t see that notation before. And yes, looks cryptic.

Re: Maximum likelihood estimation and loss functions

#25

Post by radarsat1 »

anon946 wrote: post #19

My favorite MLE example: Suppose you walk into a bank and ask them to give you a quarter. You flip the quarter twice and get two heads. Given this experiment, what do you estimate to be the probability p of getting a heads when you flip this coin? Using MLE, you would get p = 1. In other words, this coin will always give you a heads when you flip it! (According to MLE.)

Are you just demonstrating overfitting when estimating using too little data? Or is there something deeper going on in your example? What does the bank have to do with anything?

Re: Maximum likelihood estimation and loss functions

#26

Post by dumb1224 »

alalv wrote: post #21
dumb1224 wrote: post #20

Earlier quoted context omitted.

I was doing machine learning but never dig into stats before. Then I tried to study Bayesian inference and regression by myself and finally I got what it really means and its importance. First I realised that ubiquity of 'likelihood' and 'likelihood function', then I realised it's just a way to parameterise the model parameters instead of input data. Then MLE is a way to get an estimate of maximum of that function, w…

It is not the most likely setting to give rise to the data observed (that is the posterior), is the setting in which the data observed is the most likely.

Sorry my English in that sentence is probably flawed. I somehow still can't tell the difference very much.

Re: Maximum likelihood estimation and loss functions

#27

Post by Myrmornis »

unnah wrote: post #23
Myrmornis wrote: post #10

Earlier quoted context omitted.

One initial thing to understand is that the probability mass/density functions that you get taught in connection with standard probability distributions (binomial, Normal, etc) are functions of the data values : you put in a data value and the function outputs a probability (density), for some fixed parameter values. At first glance likelihood functions might look the same, but you have to think of them as functions…

An important point here is that the integral of the likelihood function over different parameter values is not constrained to be 1. This is why a likelihood is not a probability or a probability density, but its own thing. The confusing bit is that the likelihood formula is exactly the same as the formula of the original probability density function...

Splitting hairs probably but personally I'd say it the other way around: a likelihood is not a probability or a probability density, so there's no reason to think that it would integrate to 1.

The reason it's not a probability or probability density is that it's not defined to be one (in fact its definition involves a potentially different probability density for each point in parameter space).

But I think I know what you're saying -- people need to understand that it's not a probability density in order to avoid making naive probabilistic statements about parameter estimates or confidence regions when their calculations haven't used a prior over the parameters.

Re: Maximum likelihood estimation and loss functions

#28

Post by exhibit001 »

patrick451 wrote: post #17

If you have a parametrized functions that imperfectly models a real phenomenon, of course there are errors. Why assume they are random? A better assumption is that your model is just poor. Assuming deterministic modeling errors are due to randomness has always struck me as bizarre.

In the context of MLE, random has a formal definition. What you describe as poor would be included in the mathematics as a factor outside the deterministic parameters that are modeled. E.g. Y = aFactor1 + bFactor2 + ... + constant + 'poor model correction factor'.

To solve the equation, we have to make assumptions of the poor correction factor. These assumptions about the error generally have some 'mathematically nice' qualities. For example it's not predictable or has a trend relating to any other factors. An concrete example is having a mean of zero. If it had a non-zero mean, it should be accounted in the constant factor of the model.

All these mathematically nice assumptions can be summed up be calling the 'poor model correction' factor as random.

Re: Maximum likelihood estimation and loss functions

#29

Post by skzv »

sidr wrote: post #3
skzv wrote: post #2

To bring things full circle: the cross-entropy loss is the KL divergence. So intuitively, when you're minimizing cross-entropy loss, you're trying to minimize the "divergence" between the true distribution and your model distribution. This intuition really helped me understand CE loss.

Cross-entropy is not the KL divergence. There is an additional term in cross-entropy which is the entropy of the data distribution (i.e., independent of the model). So, you're right in that minimizing one is equivalent to minimizing the other. https://stats.stackexchange.com/questions/357963/what-is-the...

Yes, you are totally correct, but I believe this term is omitted from the cross-entropy loss function that is used in machine learning? Because it is a constant which does not contribute to the optimization.

Please correct me if I'm wrong.

Re: Maximum likelihood estimation and loss functions

#30

Post by anon946 »

truculent wrote: post #22
anon946 wrote: post #19

My favorite MLE example: Suppose you walk into a bank and ask them to give you a quarter. You flip the quarter twice and get two heads. Given this experiment, what do you estimate to be the probability p of getting a heads when you flip this coin? Using MLE, you would get p = 1. In other words, this coin will always give you a heads when you flip it! (According to MLE.)

The example only seems ridiculous because you've deliberately excluded relevant knowledge about the world from the model. Add a prior to the model and you'll have a much more reasonable function to maximise.

Yes, but then it is no longer MLE, but rather MAP: [https://www.cs.cmu.edu/~aarti/Class/10701_Spring23/Lecs/Lect....
Post reply on HN