Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does a least squares fit appear to have a bias when applied to simple data?

stats.stackexchange.com

Post reply on HN
Print view View original

41–50 of 84 posts

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#41

Post by defrost »

10000truths wrote: post #30
dllu wrote: post #4

You can think of it as: linear regression models only noise in y and not x, whereas ellipse/eigenvector of the PCA models noise in both x and y.

Is there any way to improve upon the fit if we know that e.g. y is n times as noisy as x? Or more generally, if we know the (approximate) noise distribution for each free variable?

> Or more generally, if we know the (approximate) noise distribution for each free variable?

This was a thing 30 odd years ago in radiometric spectrometry surveying.

The X var was time slot, a sequence of (say) one second observation accumulation windows, the Yn vars were 256 (or 512, etc) sections of the observable ground gamma ray spectrum (many low energy counts from the ground, Uranium, Thorium, Potassium, and associated breakdown daughter products; some high energy counts from the infinite cosmic background that made it through the radiation belts and atmosphere to near surface altitudes)

There was a primary NASVD (Noise Adjusted SVD) algorithm (Simple var adjustment based on expected gamma event distributions by energy levels) and a number of tweaks and variations based on how much other knowledge seemed relevant (broad area geology and radon expression by time of day, etc)

See, eg: Improved NASVD smoothing of airborne gamma-ray spectra Minty / McFadden (1998) - https://connectsci.au/eg/article-abstract/29/4/516/80344/Imp...

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#42

Post by bgbntty2 »

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different results. Which seems like a big deal because statistics is used to explain important things and to guide our life wrt those important things. What makes squares the best? I can't recall other times I've seen squares used, as my memories of my statistics training is quite blurry now, but they seem to pop up here and there in statistics relatively often, it seems.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#44

Post by djaouen »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

Squares are preferred because that is the same as minimizing Euclidean Distance, which is defined as sqrt((x2-x1)^2).

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#45

Post by stirfish »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

I haven't done it in a while, but you can do cubes (and more) too. Cubes would be the L3 norm, something about the distance between circles (spheres?) in 3d space? I need to read about norms again to tell you why or when to choose that, but I know the Googlable term is "vector norms"

I remember one is Manhattan distance, next is as-the-crow-flies straight line distance, next is if you were a crow on the earth that can also swim in a straight line underwater, and so on

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#46

Post by qnleigh »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

Least squares is guaranteed to be convex [0]. At least for linear fit functions there is only one minimum and gradient descent is guaranteed to take you there (and you can solve it with a simple matrix inversion, which doesn't even require iteration).

Intuitively this is because a multidimensional parabola looks like a bowl, so it's easy to find the bottom. For higher powers the shape can be more complicated and have multiple minima.

But I guess these arguments are more about making the problem easy to solve. There could be applications where higher powers are worth the extra difficulty. You have to think about what you're trying to optimize.

[0] https://math.stackexchange.com/questions/483339/proof-of-con...

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#47

Post by oakridge »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

One way to think of it is that each point in your data follows your model but with gaussian iid noise shifting them away. The likelihood is then product of gaussians mean shifted and rescaled by variance. Minimize the log-likelihood then becomes reducing the sum of (x-mu)^2 for each point, which is essentially least squares.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#48

Post by sfilmeyer »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

Wikipedia has some notes on why least squares, and how you might get there from other assumptions: https://en.wikipedia.org/wiki/Least_squares#Statistical_test... .

Also, quadratics are just much easier to work with in a lot of ways than higher powers. Like you said, even powers have the advantage over odd powers of not needing any sort of absolute value, but quartic equations of any kind are much harder to work with than quadratics. A local optimum on a quartic isn't necessarily a global optimum, you lose the solvability advantages of having linear derivatives, et cetera.

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#49

Post by yyhhsj0521 »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

To put it very simplistically, from mostly a practical view: abs: cannot be differentiated around 0; has multiple minima; the error space has sharp ridges power 4: way too sensitive to noise power 3: var(x+y) != var(x) + var(y)

Re: Why does a least squares fit appear to have a bias when applied to simple data?

#50

Post by paulfharrison »

bgbntty2 wrote: post #42

I haven't dealt with statistics for a while, but what I don't get is why squares specifically? Why not power of 1, or 3, or 4, or anything else? I've seen squares come up a lot in statistics. One explanation that I didn't really like is that it's easier to work with because you don't have to use abs() since everything is positive. OK, but why not another even power like 4? Different powers should give you different r…

For linear models, least squares leads to the BLUE estimator: Best Linear Unbiassed Estimator. This acronym is doing a lot of work with each of the words having a specific technical meaning.

Fitting the model is also "nice" mathematically. It's a convex optimization problem, and in fact fairly straightforward linear algebra. The estimated coefficients are linear in y, and this also makes it easy to give standard errors and such for the coefficients!

Also, this is what you would do if you were doing Maximum Likelihood assuming Gaussian distributed noise in y, which is a sensible assumption (but not a strict assumption in order to use least squares).

Also, in a geometric sense, it means you are finding the model that puts its predictions closest to y in terms of Euclidean distance. So if you draw a diagram of what is going on, least squares seems like a reasonable choice. The geometry also helps you understand things like "degrees of freedom".

So, may overlapping reasons.

Post reply on HN