Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why squared error? (2014)

benkuhn.net

Post reply on HN
Print view View original

81–90 of 101 posts

Re: Why squared error? (2014)

#81

Post by mtzet »

srean wrote: post #69
bo1024 wrote: post #64

Earlier quoted context omitted.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats. Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later. Even at that time when Gauss…

> Gauss quite openly admitted that the choice was borne out of convenience.

That's quite interesting. Do you have a reference for that?

From my understanding, the popularity of the least squares method came (at least in part) from Gauss' successful prediction of the position of Ceres. Was this just because people not using least squares were not able to calculate it?

Re: Why squared error? (2014)

#82

Post by amelius »

tomp wrote: post #78
shawnz wrote: post #36

I am no math expert, but I have always thought about it like this. The squared error is like weighting the error by the error. This causes one big error to be more significant than many small errors, which is usually what you want. Am I on the right track?

No, that's exactly why absolute error is better. "Big errors" are called outliers, they're (relatively) rare, often caused by bad data (measurement errors, typos, etc.) and substiantially influence the outcome of your calculation. In other words, squared error is less robust . But squared error is easier to compute. So, in practice, what you do is you remove outliers (e.g. cap the data at +-3sigma) then use squared e…

> So, in practice, what you do is you remove outliers (e.g. cap the data at +-3sigma) then use squared error.

But if you are say fitting a function to the data, you can't tell beforehand which data-points are the outliers. So in that case perhaps you need an iterative approach of removing them (?)

Re: Why squared error? (2014)

#83

Post by Loic »

srean wrote: post #69
bo1024 wrote: post #64

Earlier quoted context omitted.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats. Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later. Even at that time when Gauss…

3 times yes: "The latter is to be preferred when the distribution [...] is corrupted by outliers."

I am working in chemoinformatics, the main methods used by the academics to regress parameters have not changed in the past 40 years even so we went from small carefully assessed data sets (think 200 experimental points) to larger (10000, sometimes millions) with a lot of outliers from data entry errors, experimental errors, etc.

The end results is that when I see models of interest without the raw data, I reregress the parameters using my own datasets because most of the time you can barely trust them (even if coming from well known research centres).

Re: Why squared error? (2014)

#84

Post by srean »

mtzet wrote: post #81
srean wrote: post #69

Earlier quoted context omitted.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats. Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later. Even at that time when Gauss…

> Gauss quite openly admitted that the choice was borne out of convenience. That's quite interesting. Do you have a reference for that? From my understanding, the popularity of the least squares method came (at least in part) from Gauss' successful prediction of the position of Ceres. Was this just because people not using least squares were not able to calculate it?

I have come across his quote about convenience in many places, but don't have a specific reference. Perhaps The Google can help.

The other useful resource is "The Unicorn, The Normal Curve, And Other Improbable Creatures"

Re: Why squared error? (2014)

#85

Post by graycat »

srean wrote: post #71
graycat wrote: post #15

I asked that early in my career. We want a metric essentially because if we converge or have a good approximation in the metric then we are close in some important respects. Squared error, then, gives one such metric. But for some given data, usually there are several metrics we might use, e.g., absolute error (L^1), worst case error (L^infinity), L^p for positive integer p, etc. From 50,000 feet up, the reason for u…

Ah but square error is not a metric, its square root is a metric. Many nice properties of the square loss (in fact un-fucking-believably nice properties) stem not from the fact that its square root is a metric but from the fact that it is a Bregman divergence. Another oft used 'divergence' in this class is KL divergence or cross-entropy. Bregman introduced this class purely as a machinery to solve convex optimization…

> Ah but square error is not a metric, its square root is a metric.

Yes, I was using "squared error" because the OP was. What I wrote was modulo a square root missing here and there!

Re: Why squared error? (2014)

#86

Post by neutralid »

srean wrote: post #71
graycat wrote: post #15

I asked that early in my career. We want a metric essentially because if we converge or have a good approximation in the metric then we are close in some important respects. Squared error, then, gives one such metric. But for some given data, usually there are several metrics we might use, e.g., absolute error (L^1), worst case error (L^infinity), L^p for positive integer p, etc. From 50,000 feet up, the reason for u…

Ah but square error is not a metric, its square root is a metric. Many nice properties of the square loss (in fact un-fucking-believably nice properties) stem not from the fact that its square root is a metric but from the fact that it is a Bregman divergence. Another oft used 'divergence' in this class is KL divergence or cross-entropy. Bregman introduced this class purely as a machinery to solve convex optimization…

This is very interesting. Thank you.

What book would you recommend for this discussion?

Re: Why squared error? (2014)

#87

Post by srean »

neutralid wrote: post #86
srean wrote: post #71

Earlier quoted context omitted.

Ah but square error is not a metric, its square root is a metric. Many nice properties of the square loss (in fact un-fucking-believably nice properties) stem not from the fact that its square root is a metric but from the fact that it is a Bregman divergence. Another oft used 'divergence' in this class is KL divergence or cross-entropy. Bregman introduced this class purely as a machinery to solve convex optimization…

This is very interesting. Thank you. What book would you recommend for this discussion?

I am not aware of a book that talks about these things. Its spread across several research publications. Some of the older ones in Russian, but translations are available.

Re: Why squared error? (2014)

#88

Post by riskneural »

mtzet wrote: post #81
srean wrote: post #69

Earlier quoted context omitted.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats. Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later. Even at that time when Gauss…

> Gauss quite openly admitted that the choice was borne out of convenience. That's quite interesting. Do you have a reference for that? From my understanding, the popularity of the least squares method came (at least in part) from Gauss' successful prediction of the position of Ceres. Was this just because people not using least squares were not able to calculate it?

It's in the original paper in which he derives the normal distribution. Well worth a read. I last had a copy of it in the fourth basement down in the university library about fifteen years ago - it might be still there.

Re: Why squared error? (2014)

#89

Post by Sean1708 »

thomasahle wrote: post #33

It's fine to list some reasons for using squared error, but you really can't decide on the error function without referring to a problem you're trying to solve. Just look at the success of compressed sensing, based on taking the absolute value error seriously.

Which is basically the entire message of the last section.

Re: Why squared error? (2014)

#90

Post by frankc »

srean wrote: post #69
bo1024 wrote: post #64

Earlier quoted context omitted.

I don't think this is any more convincing than the article's reasons. There are closed forms to lots of things that aren't interesting.

I cannot speak for eanzenberg but I think his comment was less about his personal justification and more about the rationalizations that have been used in the history of stats. Gauss quite openly admitted that the choice was borne out of convenience. The justification using Normal or Gaussian distribution came later and the Gauss Markov result on conditional distribution came even later. Even at that time when Gauss…

Not disagreeing with your points about L1 but I want to point out that you can also do things to make L2 more robust to outliers (and have better empirical performace), such as winsorizing the data.
Post reply on HN