Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linear Regression

simonwardjones.co.uk

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Linear Regression

#61

Post by pps43 »

em500 wrote: post #55
quietbritishjim wrote: post #52

Earlier quoted context omitted.

Oops, you're absolutely right. I remember back at secondary school a teacher going through exactly the process I described, with him asking us what the best choice of line would be. The first choice someone (not me sadly) suggested was indeed this one. For others that, like me, don't already know it (at least by name): the Deming cost function is the sum of perpendicular distances to the points from the line, as oppo…

Deming regressions have some serious drawbacks compared to OLS, most importantly that it's not scale/unit-invariant: if you rescale your (e.g. use house area in square metres instead of square feet to predict house prices) you get different results. This may be less of an issue for the machine leaner who's only interested in point predictions, but it's a serious concern for us old fashioned statisticians who are more…

I would not call it a drawback. In regular regression you make an assumption that x is known exactly, and all errors are in y. In Deming regression you make another assumption, about the ratio of their variances (δ). If you change units, you need to change δ as well, and then there's no difference.

Re: Linear Regression

#63

Post by tomrod »

quietbritishjim wrote: post #52
pps43 wrote: post #42

Earlier quoted context omitted.

> that's really the most obvious cost function To me Deming regression is the most obvious. It actually took me a long time to realize that x~y and y~x are in most cases different lines.

Oops, you're absolutely right. I remember back at secondary school a teacher going through exactly the process I described, with him asking us what the best choice of line would be. The first choice someone (not me sadly) suggested was indeed this one. For others that, like me, don't already know it (at least by name): the Deming cost function is the sum of perpendicular distances to the points from the line, as oppo…

This is also called Total Least Squares -- IIRC it was developed for use when you have measurement error in X (least squares assumes no measurement error in X, all error is in Y).

Re: Linear Regression

#64

Post by tomrod »

bonoboTP wrote: post #48
ellisv wrote: post #37

Earlier quoted context omitted.

> Depends on the tradition you are coming from. Statistics :)

Yes, the statistics and machine learning research communities traditionally have a very different view of the world, of what is important, what is superfluous, but lately the two are more and more merging.

Elements of Statistical Learning is a common entry point. I'd argue they are much closer than appears.

As a subfield in computer science, of course the concern has often been on algorithmic complexity and similar. But that is nascency exposed, in my view, and likely not representative of a fully mature field.

Armchair thought (not a historian of economics): I think Econometrics followed (and continues to follow) a similar evolution -- start with the goals (identification of model parameters, identifiability, KPIs), improve statistical validity and relevance, annotate dead ends or less common routes, continue on trucking.

Re: Linear Regression

#65

Post by autokad »

em500 wrote: post #58
autokad wrote: post #56

Earlier quoted context omitted.

i wouldn't dare to say its a 'serious drawback'. as soon as you add ridge to regression (something very common for statisticians to do), it's also no longer scale/unit-invariant. Edit: this is also not just true for ridge, but lasso as well

Personally I consider the lack of scale-invariance one of the main drawbacks of most common regularizers too. Again, not a big deal if all you're after are y-hat, a bit more concerning if you're interested in beta-hat.

i try not to stick too religiously to coefficient interpretation unless its a very simple and known problem. one missing variable could change the coefficients sign.

Re: Linear Regression

#66

Post by zwaps »

autokad wrote: post #65
em500 wrote: post #58

Earlier quoted context omitted.

Personally I consider the lack of scale-invariance one of the main drawbacks of most common regularizers too. Again, not a big deal if all you're after are y-hat, a bit more concerning if you're interested in beta-hat.

i try not to stick too religiously to coefficient interpretation unless its a very simple and known problem. one missing variable could change the coefficients sign.

Depends on whether the missing variable is correlated with the beta hats you care about ;-)

Re: Linear Regression

#68

Post by zwaps »

hansvm wrote: post #50
ojnabieoot wrote: post #15

Two comments here, and I am sorry if they come across as mean: 1) I know this article isn't aimed at me, but I do truly hate the excessive emojis. 2) More substantively: not once does the word "statistics" enter here. There is not a single illustration of the idea behind linear regression, which is staggeringly simple: it is just finding the best linear fit. Anyone who looks at a 2D scatterplot can do an approximate…

Stats are important, but aren't they somewhat excessive for explaining the "idea behind linear regression"? It's neat that this implementation is an unbiased estimator in the presence of gaussian noise, but statistical reasoning seems wholly unnecessary for getting _some_ line of best fit. I bet somebody could write an article much like this and gloss over most (maybe all?) of the math: 1) Here's a 2D scatterplot, an…

The conditional mean E(y|x), which exists under quite general conditions, is the best predictor in the conventional sense of MSE.

The linear projection, assuming finite second moments and a positive definite expectation of the design matrix, is the best linear predictor in the sense of MSE. Its error is by construction (not by assumption) uncorrelated with the regressands.

The linear regression comes in when one additionally assumes that the conditional expectation of the error given the explanatory variables is zero. This then gives unbiased estimates. No distributions required.

To characterize the distributions and do testing/intervals, one usually employs large sample theory. This also then shows when the probability limit of the estimator converges to the true value (consistency) and at which speed.

The idea of requiring Gaussian noise is, except for edge cases, depreciated. Textbooks that build on this should not be used anymore. It is misleading and it leads to the wide spread misconception that normality of the error term is in some sense essential for linear regression.

It's fine to come from the CS angle of algorithms and optimizations, as opposed to theoretical distributions and all that, but semi-parametric statistics in quite useful to understand why and when linear regression can do really well.

Re: Linear Regression

#70

Post by windsignaling »

ojnabieoot wrote: post #15

Two comments here, and I am sorry if they come across as mean: 1) I know this article isn't aimed at me, but I do truly hate the excessive emojis. 2) More substantively: not once does the word "statistics" enter here. There is not a single illustration of the idea behind linear regression, which is staggeringly simple: it is just finding the best linear fit. Anyone who looks at a 2D scatterplot can do an approximate…

Linear regression isn't exclusive to statistics.

One can, for example, find a chapter on linear regression in most machine learning textbooks, without reference to statistics.

I've been on both sides of the fence, but it's always the statisticians who take the perspective of, "yeah, well you didn't study model diagnostics, ANOVA, QQ plots, ... so you don't know linear regression".

Linear regression as described in this post (which is pretty much what you find in Andrew Ng's famed machine learning course) can be used as an introductory stepping stone to logistic regression, neural networks, etc.

Post reply on HN