Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does a least squares fit appear to have a bias when applied to simple data?

stats.stackexchange.com

Post reply on HN
Print view View original

71–80 of 84 posts

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#71

Post by lambdaone »

charlieyu1 wrote: post #3

If you plot the regression line of y against x, and also x against y, you would get two different lines. I found it in the middle of teaching a stats class, and feel embarrassed. I guess normalising is one way to remove the bias.

You are absolutely correct that the difference between y against x and x against y fitting perfectly demonstrates why the bias exists, but the correct way to remove the bias is not normalization, but to use a coordinate-independent regression technique.

See the other comments by many other commenters for details.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#72

Post by srean »

emmelaich wrote: post #32
tomp wrote: post #13

Linear Regression a.k.a. Ordinary Least Squares assumes only Y has noise, and X is correct. Your "visual inspection" assumes both X and Y have noise. That's called Total Least Squares.

Yep, to demonstrate, tilt it (swap x and y) and do it again. Maybe this is what TLS does?

>(swap x and y) and do it again.

This is a great diagnostic check for symmetry.

> Maybe this is what TLS does?

No, swapping just exchanges the relation. What one needs to do is to put the errors in X and errors in Y in equal footing. That's exactly what TLS does.

Another way to think about it is that the error of a point from the line is not measured as a vertical drop parallel to Y axis but in a direction orthogonal to the line (so that the error breaks up in X and Y directions). From this orthogonality you can see that TLS is PCA (principal component analysis) in disguise.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#74

Post by frumiousirc »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

L1 (abs linear difference) is useful as minimizing on it gives an approximation of minimizing on L0 (count, aka maximizing sparsity). The reason for the substitution is that L1 has a gradient and so minimization can be fast with conventional gradient descent methods while minimizing L0 is a combinatoric problem and solving that is "hard". It is also common to add an L1 term to an L2 term to bias the solution to be sparse.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#75

Post by shiandow »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

It all boils down to the fact that mean and variance give a good approximation of a probability distribution.

In the same way that things typically converge to the average they converge even more strongly to a normal distribution. So estimating noise as a normal distribution is often good enough.

The second order approximation is just really really good, and higher orders are nigh impossible to work with.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#76

Post by RA_Fisher »

theophrastus wrote: post #15

Had a QuantSci Prof who was fond of asking "Who can name a data collection scenario where the x data has no error?" and then taught Deming regression as a generally preferred analysis [1] [1] https://en.wikipedia.org/wiki/Deming_regression

The issue in that case is that OLS is BLUE, the best linear unbiased estimator (best in the sense of minimum variance). This property is what makes OLS exceptional.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#77

Post by abanana »

kleiba wrote: post #58

> So, instead, I then diagonalized the covariance matrix to obtain the eigenvector that gives the direction of maximum variance. ...as one does...

Without knowing the meaning of that level of mathematical jargon, it feels like a "reticulating splines" sort of line. Makes me want to copy it and use it somewhere.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#78

Post by ghc »

theophrastus wrote: post #15

Had a QuantSci Prof who was fond of asking "Who can name a data collection scenario where the x data has no error?" and then taught Deming regression as a generally preferred analysis [1] [1] https://en.wikipedia.org/wiki/Deming_regression

In my field, the X data error (measurement jitter) is generally <10ns, which might as well be no error.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#79

Post by scotty79 »

dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

It might be cool to train neural network by minimizing error with assumption there's noise on both inputs and outputs.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#80

Post by dllu »

10000truths wrote: post #30
dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

Is there any way to improve upon the fit if we know that e.g. y is n times as noisy as x? Or more generally, if we know the (approximate) noise distribution for each free variable?

Yeah, you can generally "whiten" the problem by scaling it in each axis until the variance is the same in each dimension. What you describe is if x and y have a covariance matrix of like

    [ σ², 0;
      0,  (nσ)² ]
but whitening also works in general for any arbitrary covariance matrix too.

[1] https://en.wikipedia.org/wiki/Whitening_transformation

Post reply on HN