Why geometric mean?, I would ask.
"Why addition?", I would ask.
Different problems, different tools. You can't ask "why geometric mean?" without referring to a specific problem you're trying to solve.
Why squared error? (2014)
31–40 of 101 posts
Re: Why squared error? (2014)
#32An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.
ML + Big Data are a specific application of statistics To to do anything beyond use tools other people have made (and never be sure whether results are meaningful or not) statistics are required Of course, to make money from the ML boom you can probably get away with coincidence and correlation
Statistics means aggregate stuff and uses simplified characteristics out of semi-structured data. ML + Big Data allows you to ask precise questions like Where? How? Which ones?
Re: Why squared error? (2014)
#33It's fine to list some reasons for using squared error, but you really can't decide on the error function without referring to a problem you're trying to solve.
Just look at the success of compressed sensing, based on taking the absolute value error seriously.
Re: Why squared error? (2014)
#34Squared error represents the underlying belief that errors in various dimensions, or errors in independent samples, are linearly independent. So they add together like orthogonal vectors, forming a vector whose length is the square root of the sum of the squares. Minimizing the square error is a way of minimizing that square root without the superfluous operation of calculating it.
Re: Why squared error? (2014)
#35There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.
You can just use subgradient descent. Nonconvex loss would pose a bigger problem.
Re: Why squared error? (2014)
#36I am no math expert, but I have always thought about it like this. The squared error is like weighting the error by the error. This causes one big error to be more significant than many small errors, which is usually what you want. Am I on the right track?
Re: Why squared error? (2014)
#37There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.
Mean absolute error is differentiable almost everywhere. Having objectives that are not differentiable, but are differentiable almost everywhere is very common - in a deep net, if you have rectified linear activations (very common) or L1 regularisation (not unheard of), you have an objective that is not differentiable everywhere ... but the methods still work.
Re: Why squared error? (2014)
#38For minimizing the square of the errors I think the good reason is because, assuming your data has gaussian probability distribution, minimizing the square error corresponds to maximizing the likelihood of the measurement, as you and others have said. Why do we assume gaussian errors? There is seldom a gaussian distribution in the real world usually because the probability for large error values doesn't not decay tha…
That's a summary of the article.
Yes, sort of. But I think he says a lot of unnecessary things not getting at the root of the issue.
I left out some detail I should have said, like what is so special about a gaussian that makes the math easy. So I will say it.
A measurement can infer a probability distribution for what the measured quantity is. A second measurement, on its own, also infers some probability distribution for what the measured quantity is. It we consider both measurements together, we get yet another probability distribution for what the measured quantity is. The magic is that if we had a gaussian distribution for the measurements, then the distribution for the combined measurements is also a gaussian. This is not true in general. As long as we have gaussian distributions we can do all the operations we want and the probability distributions are gaussian and can be fully described by a center point and a width. (Forgive me for the liberties I am taking here.) The basic alternative to exactly solving the problem is to actually try to carry around the probability distribution functions, which is not practical even with very powerful computers.
Re: Why squared error? (2014)
#39Another pro tip - absolute error magnitude is the convex hull of non-zero entry count for vectors (l_0 norm in some circles). So in the convex minimization context (and for most other smooth loss terms in general) you end up with solutions with more zero entries and few possibly large non-zero entries.
Re: Why squared error? (2014)
#40The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed.
Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.