Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does a least squares fit appear to have a bias when applied to simple data?

stats.stackexchange.com

Post reply on HN
Print view View original

31–40 of 84 posts

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#31

Post by moregrist »

theophrastus wrote: post #15

Had a QuantSci Prof who was fond of asking "Who can name a data collection scenario where the x data has no error?" and then taught Deming regression as a generally preferred analysis [1] [1] https://en.wikipedia.org/wiki/Deming_regression

Most of the time, if you have a sensor that you sample at, say 1 KHz and you’re using a reliable MCU and clock, the noise terms in the sensor will vastly dominate the jitter of sampling.

So for a lot of sensor data, the error in the Y coordinate is orders of magnitude higher than the error in the X coordinate and you can essentially neglect X errors.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#32

Post by emmelaich »

tomp wrote: post #13

Linear Regression a.k.a. Ordinary Least Squares assumes only Y has noise, and X is correct. Your "visual inspection" assumes both X and Y have noise. That's called Total Least Squares.

Yep, to demonstrate, tilt it (swap x and y) and do it again. Maybe this is what TLS does?

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#33

Post by jjk166 »

analog31 wrote: post #7
dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

That brings up an interesting issue, which is that many systems do have more noise in y than in x. For instance, time series data from an analog-to-digital converter, where time is based on a crystal oscillator.

Well yeah, x is specifically the thing you control, y is the thing you don't. For all but the most trivial systems, y will be influenced by something besides x which will be a source of noise no matter how accurately you measure. Noise in x is purely due to setup error. If your x noise was greater than your y noise, you generally wouldn't bother taking the measurement in the first place.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#34

Post by CGMthrowaway »

dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

So when fitting a trend, e.g. for data analytics, should we use eigenvector of the PCA instead of linear regression?

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#35

Post by CGMthrowaway »

sega_sai wrote: post #6

The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.

That makes sense. Why does least squares skew the line downwards though (Vs some other direction)? Seems arbitrary

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#36

Post by SubiculumCode »

em500 wrote: post #16

Sorry for my negativity / meta comment on this thread. From what I can tell the stackexchange discussion in the submission already to provides all the relevant points to be discussed about this. While the asymmetry of least squares will probably be a bit of a novelty/surprise to some, pretty much anything posted here is more or less a copy of one of the comments on stackexchange. [Challenge: provide a genuinely novel…

But bringing it up as a topic, aside from being informative, allows for more varied conversation that is allowed on stack exchange, like exploring alternative modeling approaches. It may not have happened, but the possibility can only present itself given the opportunity

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#37

Post by MontyCarloHall »

ryang2718 wrote: post #8
sega_sai wrote: post #6

The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.

I find it helpful to view least as fitting the noise to a Gaussian distribution.

They both fit Gaussians, just different ones! OLS fits a 1D Gaussian to the set of errors in the y coordinates only, whereas TLS (PCA) fits a 2D Gaussian to the set of all (x,y) pairs.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#38

Post by sega_sai »

CGMthrowaway wrote: post #35
sega_sai wrote: post #6

The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.

That makes sense. Why does least squares skew the line downwards though (Vs some other direction)? Seems arbitrary

I think it has to do with the ratio of \Sigma_xx, \Sigma_yy. I don't have time to verify that, but it should be easy to check analytically.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#39

Post by thaumasiotes »

noncovalence wrote: post #28

This problem is usually known as regression dilution, discussed here: https://en.wikipedia.org/wiki/Regression_dilution

Is it? The wikipedia article says that regression dilution occurs when errors in the x data bias the computed regression line.

But the stackexchange question is asking why an unbiased regression line doesn't lie on the major axis of the 3σ confidence ellipse. This lack of coincidence doesn't require any errors in the x data. https://stats.stackexchange.com/a/674135 gives a constructed example where errors in the x data are zero by definition.

Unless I'm misunderstanding something?

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#40

Post by ryang2718 »

MontyCarloHall wrote: post #37
ryang2718 wrote: post #8

Earlier quoted context omitted.

I find it helpful to view least as fitting the noise to a Gaussian distribution.

They both fit Gaussians, just different ones! OLS fits a 1D Gaussian to the set of errors in the y coordinates only, whereas TLS (PCA) fits a 2D Gaussian to the set of all (x,y) pairs.

Well, that was a knowledge gap, thank you! I certainly need to review PCA but python makes it a bit too easy.
Post reply on HN