Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why squared error? (2014)

benkuhn.net

Post reply on HN
Print view View original

41–50 of 101 posts

Re: Why squared error? (2014)

#41

Post by robotresearcher »

shawnz wrote: post #36

I am no math expert, but I have always thought about it like this. The squared error is like weighting the error by the error. This causes one big error to be more significant than many small errors, which is usually what you want. Am I on the right track?

> This causes one big error to be more significant than many small errors,

That's correct.

> which is usually what you want

Unless you have outliers, in which case it's what you don't want. So you add e.g. a Huber loss function to reach a compromise.

Re: Why squared error? (2014)

#42

Post by eanzenberg »

tvural wrote: post #40

The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed. Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.

Regardless of how distributed the errors are, the squared error fit will provide the expectation value of the variable, which is the mean. It will say nothing of the error of the mean it calculates.

Re: Why squared error? (2014)

#43

Post by thanatropism »

throw_away_777 wrote: post #11

There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

No it isn't.

Differentiability is important if you want to have an closed-form formula and derive it in front of undergraduates.

Re: Why squared error? (2014)

#44

Post by nightcracker »

throw_away_777 wrote: post #11

There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

What exactly would go wrong if you assume that the derivative is zero at x = 0?

And aren't exact zeroes an error scenario for most machine learning models anyway?

Re: Why squared error? (2014)

#45

Post by eanzenberg »

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form.

Why L2 regularization? Same reason. A closed form solution exists from linear algebra.

But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

Re: Why squared error? (2014)

#47

Post by fiatjaf »

thomasahle wrote: post #31
fiatjaf wrote: post #28

Why geometric mean?, I would ask.

"Why addition?", I would ask. Different problems, different tools. You can't ask "why geometric mean?" without referring to a specific problem you're trying to solve.

What is a problem geometric mean solve? That was my question the entire time.

When people ask "why machine learning?" the answers are "machine learning can do these things blablabla", not "you must specify the problem you're trying to solve".

Re: Why squared error? (2014)

#48

Post by highd »

bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

Sorry you're getting down-voted, I don't think it's an unreasonable question.

In the sense I think you're using it, "statistics" are really methods for dimensionality reduction - we take means, and medians and standard deviations with the hopes that they will capture the parts of the data we care about. This is important for two reasons - for one, for anything even moderately high dimension we'll never have enough data to be able to forego some means of aggregation due to the "curse of dimensionality". Secondly, the human-machine interaction information bandwidth is annoyingly low, so we need some way to compress any information for human consumption. "Statistics" are one way we do so.

"Statistics" is also a field of study based around understanding how multiple data points relate to each other - that is of course critical to machine learning, and I think the terminology collision is why you're getting downvoted.

Re: Why squared error? (2014)

#49

Post by highd »

tvural wrote: post #40

The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed. Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.

"if the best model is not differentiable, you should still use it."

I'm not sure I would say that - neural nets are "near everywhere differentiable", for example. Without differentiability we're stuck with, for example, discrete GAs for optimization, and you can throw all your intuition out the window (not to mention training/learning efficiency).

Re: Why squared error? (2014)

#50

Post by j1vms »

_Wintermute wrote: post #18
bitL wrote: post #16

An honest question - do we even need statistics when we have machine learning? Statistics to me appears as a hack/aggregation of data we couldn't process at once in the past; these days ML + Big Data can achieve that and instead of statistics we can do computational inference instead. To me this looks like looking back to "old ways" for a reference point instead of looking forward to the unknown but more exciting.

Yes we still need statistics. There is a huge overlap between machine learning methods and applied statistics, so much so that often there is not a clear distinction between the two.

> between machine learning methods and applied statistics (...) often there is not a clear distinction between the two.

I would say applied statistics draws a line just prior to implementation concerns (say, real-world resource usage measured in time, space and energy) whereas these would be fully within scope and of interest in machine learning.

As an example, applied statistics could provide a useful approach to a vision/image recognition problem, and this approach might be provably unrealizable in practice using real-world execution units (e.g. CUDA cores). Nonetheless, it might still be a very worthwhile theoretical result in applied statistics, although of no immediate interest within ML except to hint at potential new area of research.

Post reply on HN