Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why squared error? (2014)

benkuhn.net

Post reply on HN
Print view View original

51–60 of 101 posts

Re: Why squared error? (2014)

#51

Post by gabrielgoh »

highd wrote: post #49
tvural wrote: post #40

The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed. Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.

"if the best model is not differentiable, you should still use it." I'm not sure I would say that - neural nets are "near everywhere differentiable", for example. Without differentiability we're stuck with, for example, discrete GAs for optimization, and you can throw all your intuition out the window (not to mention training/learning efficiency).

A few misconceptions I should correct in this comment.

- There is plenty of existing technology for handling non-differentiable function. Functions like the absolute value, 2-norm, and so on have a generalization of the gradient (the subgradient) which can be used in lieu of the gradient.

- That functions are "almost everywhere differentiable" (i.e. the non-differentability lies in a manifold of zero measure) makes these functions behave pretty much like smooth ones. This is often not the case as optima often conspire to lie exactly on these nonsmooth manifolds.

Re: Why squared error? (2014)

#52

Post by lottin »

eanzenberg wrote: post #45

Why squared error? Because you can solve the equation to minimize squared error using linear algebra in closed form. Why L2 regularization? Same reason. A closed form solution exists from linear algebra. But at the end of the day, you are most interested in the expectation value of the coefficient and minimizing the squared error gives you E[coeffs] which is the mean of the coefficients.

> Because you can solve the equation to minimize squared error using linear algebra in closed form.

Exactly right. It has nothing to do with probability distributions.

Re: Why squared error? (2014)

#53

Post by thisrod »

Squared error because the uncertainties in independent, normally distributed random variables add in quadrature. I expect that this could be proved geometrically using Pythagoras's theorem, so in that sense the comments about orthogonal axes are vaguely on the right track.

Normally distributed variables because the central limit theorem.

It isn't all that complicated.

Re: Why squared error? (2014)

#54

Post by kkylin »

gabrielgoh wrote: post #51
highd wrote: post #49

Earlier quoted context omitted.

"if the best model is not differentiable, you should still use it." I'm not sure I would say that - neural nets are "near everywhere differentiable", for example. Without differentiability we're stuck with, for example, discrete GAs for optimization, and you can throw all your intuition out the window (not to mention training/learning efficiency).

A few misconceptions I should correct in this comment. - There is plenty of existing technology for handling non-differentiable function. Functions like the absolute value, 2-norm, and so on have a generalization of the gradient (the subgradient) which can be used in lieu of the gradient. - That functions are "almost everywhere differentiable" (i.e. the non-differentability lies in a manifold of zero measure) makes t…

And error measures involving sum of absolute values (i.e., L1 norm) are central to methods like lasso (https://en.wikipedia.org/wiki/Lasso_(statistics)) and their cousins.

Re: Why squared error? (2014)

#55

Post by highd »

gabrielgoh wrote: post #51
highd wrote: post #49

Earlier quoted context omitted.

"if the best model is not differentiable, you should still use it." I'm not sure I would say that - neural nets are "near everywhere differentiable", for example. Without differentiability we're stuck with, for example, discrete GAs for optimization, and you can throw all your intuition out the window (not to mention training/learning efficiency).

A few misconceptions I should correct in this comment. - There is plenty of existing technology for handling non-differentiable function. Functions like the absolute value, 2-norm, and so on have a generalization of the gradient (the subgradient) which can be used in lieu of the gradient. - That functions are "almost everywhere differentiable" (i.e. the non-differentability lies in a manifold of zero measure) makes t…

Yes, that was what I was saying. Absolute value, 2-norm are fine thanks to subgradient techniques and theory, as well as their differentiability over the majority of the function - but you can imagine tons of non-differentiable models where the subgradient is mostly useless and we generally use convex relaxations or other smoother analogs.

I don't think there was any misconception.

Re: Why squared error? (2014)

#56

Post by TeMPOraL »

My explanation for squared error in linear approximation always was: because it minimizes the thickness of the line that passes through all the data points.

(Per the old math joke - you can make a line passing through any three points on a plane if you make it thick enough.)

Re: Why squared error? (2014)

#57

Post by throw_away_777 »

thanatropism wrote: post #43
throw_away_777 wrote: post #11

There is a Kaggle competition right now that uses mean absolute error, and this makes the problem substantially harder. For a practical discussion of techniques used to solve machine learning problems that use mae see the forums in: https://www.kaggle.com/c/allstate-claims-severity/forums As touched upon in the article, the objective not being differentiable is a big deal for modern machine learning methods.

No it isn't. Differentiability is important if you want to have an closed-form formula and derive it in front of undergraduates.

This is the difference between practice and theory. In theory differential objectives don't matter, in practice for medium to large datasets they make machine learning a lot faster. Speed is critical, as you need to be able to iterate quickly. The solution most commonly used on Kaggle is to transform the target feature and then minimize mean squared error, but there is some systematic uncertainty introduced by this.

Re: Why squared error? (2014)

#58

Post by throw_away_777 »

tvural wrote: post #40

The best explanation is probably that squared error gives you the best fit when you assume your errors should normally distributed. Things like the fact that squared error is differentiable are actually irrelevant - if the best model is not differentiable, you should still use it.

The fact that squared error is differentiable is not irrelevant. You can solve some machine learning models faster with differentiable objectives (most notably xgboost). Speed is important, you need to optimize your models and the longer it takes to run a model the less things you can try.

Re: Why squared error? (2014)

#59

Post by 443633456 »

bitL wrote: post #30
klodolph wrote: post #26

Earlier quoted context omitted.

Machine learning is often considered a statistical technique. The main difference seems to be that in traditional statistics, people derive practice from theory, whereas in ML people will try out techniques and figure out the theory later. That's really just a cultural difference. The techniques for analyzing ML models are all statistical to begin with. Statistics, as a field, already used general-purpose optimizatio…

There is much more in ML than just statistics. I was basically asking why the "statistics filter" is so often on in ML. Neural networks don't seem a statistical technique, even if somebody uses them for regression. Yes, there is an overlap, but no, ML != statistics. As you mentioned, non-linear optimization is used in statistics on meta-level however nobody claims statistics is operations research or vice versa.

I agree with this comment because ML comprises some techniques that are still not understood [1].

I would like to think that statistics comes more from a pure math approach, loosely, while ML comes from an applied math approach, loosely. ML works spectacularly well on a class of problems. Why it does what it does is (in)conveniently brushed under the rug. How you treat that (in)convenience is left to you.

[1] - https://www.quantamagazine.org/20151203-big-datas-mathematic...

Re: Why squared error? (2014)

#60

Post by bzbarsky »

fiatjaf wrote: post #47
thomasahle wrote: post #31

Earlier quoted context omitted.

"Why addition?", I would ask. Different problems, different tools. You can't ask "why geometric mean?" without referring to a specific problem you're trying to solve.

What is a problem geometric mean solve? That was my question the entire time. When people ask "why machine learning?" the answers are "machine learning can do these things blablabla", not "you must specify the problem you're trying to solve".

> What is a problem geometric mean solve?

It gives you a way to average together two things that have units that have nothing to do with each other and then compare two such averages and have the comparison make sense, as long as your units were consistent.

As a silly example, say you want to average 1kg and 1m and compare that average to the average 2kg and 0.5m. With arithmetic mean, ignoring the fact that it's nonsense to add different units, you could get numbers like (1+1)/2 = 1 and (2+0.5)/2 = 1.25 if you use kg and m, but numbers like (1 + 100)/2 = 50.5 and (2 + 50)/2 = 26 if you use kg and cm. Notice that which one is bigger depends on your choice of units. On the other hand, the geometric mean of the two examples is always the same as long as you use consistent units: 1 for both if you use kg and m, and 10 for both if you use kg and cm.

In practice, this sort of operation is only useful if you have multiple measures of some sort along different axes (think performance on 3 different performance tests) and you're being forced to produce a single average number. Again, a fairly silly thing to do, but _very_ common: just about every single performance benchmark does this.

Post reply on HN