Why does a least squares fit appear to have a bias when applied to simple data?
stats.stackexchange.com
Why does a least squares fit appear to have a bias when applied to simple data?
1–10 of 84 posts
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#2[dead]
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#3If you plot the regression line of y against x, and also x against y, you would get two different lines.
I found it in the middle of teaching a stats class, and feel embarrassed.
I guess normalising is one way to remove the bias.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#4You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#5[dead]
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#6The least squares and pca minimize different loss functions. One is
sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#7You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.
That brings up an interesting issue, which is that many systems do have more noise in y than in x. For instance, time series data from an analog-to-digital converter, where time is based on a crystal oscillator.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#8The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.
I find it helpful to view least as fitting the noise to a Gaussian distribution.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#9You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.
That brings up an interesting issue, which is that many systems do have more noise in y than in x. For instance, time series data from an analog-to-digital converter, where time is based on a crystal oscillator.
This fact underlies a lot of causal inference.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#10The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.
I find it helpful to view least as fitting the noise to a Gaussian distribution.
OLS estimator is the minimum-variance linear unbiased estimator even without the assumption of Gaussian distribution.