Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does a least squares fit appear to have a bias when applied to simple data?

stats.stackexchange.com

Post reply on HN
Print view View original

61–70 of 84 posts

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#61

Post by bravura »

jjk166 wrote: post #33
analog31 wrote: post #7

Earlier quoted context omitted.

That brings up an interesting issue, which is that many systems do have more noise in y than in x. For instance, time series data from an analog-to-digital converter, where time is based on a crystal oscillator.

Well yeah, x is specifically the thing you control, y is the thing you don't. For all but the most trivial systems, y will be influenced by something besides x which will be a source of noise no matter how accurately you measure. Noise in x is purely due to setup error. If your x noise was greater than your y noise, you generally wouldn't bother taking the measurement in the first place.

“ If your x noise was greater than your y noise, you generally wouldn't bother taking the measurement in the first place.”

Why not? You could still do inference in this case.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#62

Post by porridgeraisin »

rjdj377dhabsn wrote: post #12
LudwigNagasena wrote: post #10

Earlier quoted context omitted.

OLS estimator is the minimum-variance linear unbiased estimator even without the assumption of Gaussian distribution.

Yes, and if I remember correctly, you get the Gaussian because it's the minimum entropy (least additional assumptions about the shape) continuous distribution given a certain variance.

And given a mean.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#63

Post by stdbrouw »

CGMthrowaway wrote: post #34
dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

So when fitting a trend, e.g. for data analytics, should we use eigenvector of the PCA instead of linear regression?

(Generalized) linear models have a straightforward probabilistic interpretation -- E(Y|X) -- which I don't think is true of total least squares. So it's more of an engineering solution to the problem, and in statistics you'd be more likely to go for other methods such as regression calibration to deal with measurement error in the independent variables.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#64

Post by lucketone »

djaouen wrote: post #44
bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

Squares are preferred because that is the same as minimizing Euclidean Distance, which is defined as sqrt((x2-x1)^2).

sqrt((x2-x1)^2) == x2-x1

I think you meant sqrt(x^2+y^2)

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#65

Post by mr_toad »

paulfharrison wrote: post #50
bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

For linear models, least squares leads to the BLUE estimator: Best Linear Unbiassed Estimator. This acronym is doing a lot of work with each of the words having a specific technical meaning. Fitting the model is also "nice" mathematically. It's a convex optimization problem, and in fact fairly straightforward linear algebra. The estimated coefficients are linear in y, and this also makes it easy to give standard erro…

Best meaning the ‘least variance’, where variance is calculated based on the sum of squared residuals. There is a circularity in that definition.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#66

Post by sigmoid10 »

moregrist wrote: post #31
theophrastus wrote: post #15

Had a QuantSci Prof who was fond of asking "Who can name a data collection scenario where the x data has no error?" and then taught Deming regression as a generally preferred analysis [1] [1] https://en.wikipedia.org/wiki/Deming_regression

Most of the time, if you have a sensor that you sample at, say 1 KHz and you’re using a reliable MCU and clock, the noise terms in the sensor will vastly dominate the jitter of sampling. So for a lot of sensor data, the error in the Y coordinate is orders of magnitude higher than the error in the X coordinate and you can essentially neglect X errors.

That is actually the case in most fields outside of maybe clinical chemistry and such, where Deming became famous for explaining it (despite not even inventing the method). Ordinary least squares originated in astronomy, where people tried to predict movement of celestial objects. Timing a planet's position was never an issue (in fact time is defined by celestian position), but getting the actual position of a planet was.

Total least squares regression also is highly non-trivial because you usually don't measure the same dimension on both axes. So you can't just add up errors, because the fit will be dependent on the scale you chose. Deming skirts around this problem by using the ratio of variances of errors (division also works for different units), but that is rarely known well. Deming works best when the measurement method for both dependent and independent variable is the same (for example when you regress serum levels against one another), meaning the ratio is simply one. Which of course implies that they have the same unit. So you don't run into the scale-invariance issues, which you would in most natural science fields.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#67

Post by jjk166 »

bravura wrote: post #61
jjk166 wrote: post #33

Earlier quoted context omitted.

Well yeah, x is specifically the thing you control, y is the thing you don't. For all but the most trivial systems, y will be influenced by something besides x which will be a source of noise no matter how accurately you measure. Noise in x is purely due to setup error. If your x noise was greater than your y noise, you generally wouldn't bother taking the measurement in the first place.

“ If your x noise was greater than your y noise, you generally wouldn't bother taking the measurement in the first place.” Why not? You could still do inference in this case.

You could, and maybe sometimes you would, but generally you won't. If at all possible, it makes a lot more sense to improve your setup to reduce the x noise, either with a better setup or changing your x to be something you can better control.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#68

Post by mr_toad »

CGMthrowaway wrote: post #35
sega_sai wrote: post #6

The least squares and pca minimize different loss functions. One is sum of squares of vertical(y) distances, another is is sum of closest distances to the line. That introduces the differences.

That makes sense. Why does least squares skew the line downwards though (Vs some other direction)? Seems arbitrary

The Pythagorean distance would assume that some of the distance (difference) is on the x axis, and some on the y axis, and the total distance is orthogonal to the fitted line.

OLS assumes that x is given, and the distance is entirely due to the variance in y, (so parallel to the y axis). It’s not the line that’s skewed, it’s the space.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#69

Post by Beretta_Vexee »

theophrastus wrote: post #15

Had a QuantSci Prof who was fond of asking "Who can name a data collection scenario where the x data has no error?" and then taught Deming regression as a generally preferred analysis [1] [1] https://en.wikipedia.org/wiki/Deming_regression

For most time series, noise in time measurement is negligible. However, this does not prevent complex coupling phenomena from occurring for other parameters, such as GPS coordinates.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#70

Post by Someone »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

A power of 1 doesn’t guarantee a unique solution. A simple example has 3 points:

  (0,0), (1,0), (1,1)
Any y = a × x with a between zero and one gives you a sum of errors of 1.

Powers less than 1 have the undesired property that they will prefer making one large error over multiple small ones. With the same 3-point example and a power of ½, you get:

- for y = 0, a cumulative error of 1

- for y = x/2, a cumulative error of 2 times √½. That’s √2 or about 1.4

- for y = x, a cumulative error of 1

(Underlying reason is that √(|x|+|y|) . Conversely for powers p larger than 1, we have *(|x|+|y|)^p > |x|^p + |y|^p)

Odd powers would require you to take absolute differences to avoid getting, for example, an error of -2 giving a contribution of (-2)³ = -8 to the cumulative error. Otherwise they would work fine.

The math for squares (https://en.wikipedia.org/wiki/Simple_linear_regression) is easy, even when done by hand, and has some desirable properties (https://en.wikipedia.org/wiki/Simple_linear_regression#Numer...)

Post reply on HN