[deleted]
Why does a least squares fit appear to have a bias when applied to simple data?
21–30 of 84 posts
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#22Many times I've looked at the output of a regression model, seen this effect, and then thought my model must be very bad. But then remember the points made elsewhere in thread. One way to visually check that the fit line has the right slope is to (1) pick some x value, and then (2) ensure that the noise on top of the fit is roughly balanced on either side. I.e., that the result does look like y = prediction(x) + epsi…
Maybe comparing plots of residuals makes it clearest.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#23[deleted]
or
3) there's a misunderstanding about ordinary least-squares.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#24Sorry for my negativity / meta comment on this thread. From what I can tell the stackexchange discussion in the submission already to provides all the relevant points to be discussed about this. While the asymmetry of least squares will probably be a bit of a novelty/surprise to some, pretty much anything posted here is more or less a copy of one of the comments on stackexchange. [Challenge: provide a genuinely novel…
This answer is too grown-up for the forum.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#25Sorry for my negativity / meta comment on this thread. From what I can tell the stackexchange discussion in the submission already to provides all the relevant points to be discussed about this. While the asymmetry of least squares will probably be a bit of a novelty/surprise to some, pretty much anything posted here is more or less a copy of one of the comments on stackexchange. [Challenge: provide a genuinely novel…
[deleted]
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#26Earlier quoted context omitted.
That brings up an interesting issue, which is that many systems do have more noise in y than in x. For instance, time series data from an analog-to-digital converter, where time is based on a crystal oscillator.
This fact underlies a lot of causal inference.
I’m not an SME here and would love to hear more about this.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#27The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.
I find it helpful to view least as fitting the noise to a Gaussian distribution.
Both of these do, in a way. They just differ in which gaussian distribution they're fitting to.
And how I suppose. PCA is effectively moment matching, least squares is max likelihood. These correspond to the two ways of minimizing the Kullback Leibler divergence to or from a gaussian distribution.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#28This problem is usually known as regression dilution, discussed here: https://en.wikipedia.org/wiki/Regression_dilution
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#29Sorry for my negativity / meta comment on this thread. From what I can tell the stackexchange discussion in the submission already to provides all the relevant points to be discussed about this. While the asymmetry of least squares will probably be a bit of a novelty/surprise to some, pretty much anything posted here is more or less a copy of one of the comments on stackexchange. [Challenge: provide a genuinely novel…
The stackexchange discussion already provides a good answer.
I think there is not much to be said. It is not a puzzle for us to solve, just a neat little mathematical observation.
Re: Why does a least squares fit appear to have a bias when applied to simple data?
#30You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.
Is there any way to improve upon the fit if we know that e.g. y is n times as noisy as x? Or more generally, if we know the (approximate) noise distribution for each free variable?