Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

Post reply on HN
Print view View original

81–90 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#81

Post by kyllo »

soVeryTired wrote: post #37
jbjbjbjb wrote: post #35

Earlier quoted context omitted.

I’ve always felt that ML introductions completely butcher OLS. When I was taught it in stats we had to consider the Gauss-Markov conditions and interpret the coefficients, we would study the residuals. ML introductions just focus getting good predictions.

IMO that's the fundamental difference between statistics and ML. The culture of stats is about fitting a model and interpreting the fit, while the culture of ML is to treat the model as a black box. That's one of the reasons that multicollinearity is seen as a big deal by statisticians, but ML practitioners couldn't give a hoot.

Only perfect multicollinearity (correlation of 1.0 or -1.0) is a problem at the linear algebra level when fitting a statistical model.

But theoretically speaking, in a scientific context, why would you want to fit an explanatory model that includes multiple highly (but not perfectly) correlated independent variables?

It shouldn't be an accident. Usually it's because you've intentionally taken multiple proxy measurements of the same theoretical latent variable and you want to reduce measurement error. So that becomes a part of your measurement and modeling strategy.

Re: How linear regression works intuitively and how it leads to gradient descent

#82

Post by esafak »

jampekka wrote: post #75
esafak wrote: post #69

Earlier quoted context omitted.

...because stochastic methods are implicit regularizers, leading to solutions that generalize better. Let's spell it out for those that don't know. https://www.inference.vc/notes-on-the-origin-of-implicit-reg...

OLS is a convex optimization problem, so this doesn't really apply. And for statistical analysis you really don't want to add poorly understood artificial noise to the parameter estimates anyway.

In general you do, because the unbiased estimates have higher generalization error. You are already dealing with sampling noise. I am not an expert in optimization, and what "poorly understood" means to you, but I know there is quite some research on the properties of SGD noise; e.g., https://francisbach.com/rethinking-sgd-noise/

Dissecting the Effects of SGD Noise in Distinct Regimes of Deep Learning https://arxiv.org/abs/2301.13703

Re: How linear regression works intuitively and how it leads to gradient descent

#83

Post by LPisGood »

tibbar wrote: post #78

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal…

What you’re describing is the technique known as the “kernel trick”, correct?

Re: How linear regression works intuitively and how it leads to gradient descent

#84

Post by SubiculumCode »

tibbar wrote: post #78

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal…

In my work, I hardly ever use linear regression, but do use multiple linear regression. Multiple linear regression allows multiple linear predictors, where the method parses shared and independent variances associated with each predictor. These discussions on linear regression hardly ever touches on the very useful multiple linear regression method. In the case of bad variance inflation in models with multi-collinear predictors, robust regression techniques are advised like ridge, LASSO, or elastic net regression.

In relation to gradient descent, I do not know enough if multiple regression is at all relevant, or why not.

And yeah, for non-normal error distributions, we should be looking at generalized linear models, which allows one to specify other distributions that might better fit the data.

Re: How linear regression works intuitively and how it leads to gradient descent

#85

Post by tinix »

lucasfcosta wrote: post #70
liamwire wrote: post #29

Earlier quoted context omitted.

I know this is repeated ad nauseam by now, but as an ardent user of em dashes for many years pre-LLM, I think this a bad heuristic.

Co-author and founder of Briefer here. I used to use em dashes before they were cool. I actually learned about them when I emailed a guy who's a software engineer at Genius and also writes for The New Yorker and The Atlantic. I asked him for tips on how to write well and he recommended that I read Steven Pinker's "The Sense of Style", which uses em dashes exhaustively, and explains when and why one should use them. I…

I also recommend "The Sense of Style"; knowing how to wield punctuation and grammatical structure is critical for clearly and successfully articulating your ideas. I use semicolons, colons, and parentheticals heavily (but en dashes and em dashes are great too).

vs

I also recommend "The Sense of Style"--knowing how to wield punctuation and grammatical structure is critical for clearly and successfully articulating your ideas--and I use semicolons, colons, and parentheticals heavily (but en dashes and em dashes are great too).

I find that dashes are great for conversational style flowing sentence structure, but sometimes they can become too long and tiring to the reader.

Re: How linear regression works intuitively and how it leads to gradient descent

#86

Post by monkeyelite »

JadeNB wrote: post #71
monkeyelite wrote: post #61

Earlier quoted context omitted.

Dont take the “for engineers” version. > and "proving" theorems mechanically I think you’ve have a bad experience because writing a proof is explaining deep understanding.

> I think you’ve have a bad experience because writing a proof is explaining deep understanding. I think your wording is the key— coming up with a proof is creating deep understanding, but writing a proof very much need not be explaining or creating deep understanding. Writing a proof can be done mechanically, by both instructor and student, and, if done so, neither demonstrates nor creates understanding. (Also, in s…

> need not shed any light on the actual practice of statistics.

That’s not what this comment asked for.

Re: How linear regression works intuitively and how it leads to gradient descent

#87

Post by [deleted] »

LPisGood wrote: post #83
tibbar wrote: post #78

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal…

What you’re describing is the technique known as the “kernel trick”, correct?

[deleted]

Re: How linear regression works intuitively and how it leads to gradient descent

#88

Post by alejohausner »

JadeNB wrote: post #71
monkeyelite wrote: post #61

Earlier quoted context omitted.

Dont take the “for engineers” version. > and "proving" theorems mechanically I think you’ve have a bad experience because writing a proof is explaining deep understanding.

> I think you’ve have a bad experience because writing a proof is explaining deep understanding. I think your wording is the key— coming up with a proof is creating deep understanding, but writing a proof very much need not be explaining or creating deep understanding. Writing a proof can be done mechanically, by both instructor and student, and, if done so, neither demonstrates nor creates understanding. (Also, in s…

You're right. Coming up with a proof is a creative process. Each major proof in mathematics is so unique, that it usually gets named after its inventor. So we have Euclid's proof that there are infinitely many primes, Euler's proof that e is irrational, and Wiles' proof of Fermat's last theorem.

Re: How linear regression works intuitively and how it leads to gradient descent

#89

Post by levocardia »

tibbar wrote: post #78

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal…

Non-constant variance does not actually bias the coefficients of a linear regression model -- thus, its predictions will be just fine. What it does is underestimate the standard errors; your p-values will typically be too small. Sometimes a log-transform or similar can help, but otherwise you can use weighted least-squares.

This kind of problem is actually a good intro to iterative refitting methods for regression models: How do you know what the weights should be? Well, you fit the initial model with no weights, get its residuals, use those to fit another model, rinse and repeat until convergence. A good learning experience and easy to hand-code.

Re: How linear regression works intuitively and how it leads to gradient descent

#90

Post by levocardia »

LPisGood wrote: post #83
tibbar wrote: post #78

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal…

What you’re describing is the technique known as the “kernel trick”, correct?

No, the kernel trick is something else: basically a nonlinear basis representation of the model. For example, fitting a polynomial model, or using splines, would effectively be using the "kernel trick" (though only ML people use that term, not statisticians, and usually they talk about it in the context of SVMs but it's fine for linear regression too). Transforming the data is just transforming the Y-outcome, most commonly with log(y) for things that tend to be distributed with a right-skew: house prices being a classic example, along with things like income, various blood biomarkers, or really anything that cannot go below zero but can (in principle) be arbitrarily large.

In a few rare cases I have found situations where sqrt(y) or 1/y is a clever and useful transform but they're very situational, often occurring when there's some physical law behind the data generation process with that sort of mathematical form.

Post reply on HN