Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

Post reply on HN
Print view View original

31–40 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#31

Post by easygenes »

BlueUmarell wrote: post #16
easygenes wrote: post #7

This is very light and approachable but stops short of building the statistical intuition you want here. They fixate on the smoothness of squared errors without connecting that to the gaussian noise model and establishing how that relates to the predictive power against natural sorts of data.

Any resource/link you know of that further develops your point?

CMU lecture notes [0] I think approach it in an intuitive way, starting from the Gaussian noise linear model, deriving log-likelihood, and presenting the analytic approach. Misses the bridge to gradient methods though.

For gradients, Stanford CS229 [1] jumps right into it.

[0] https://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/06/lectu...

[1] https://cs229.stanford.edu/lectures-spring2022/main_notes.pd...

Re: How linear regression works intuitively and how it leads to gradient descent

#32

Post by easygenes »

jfjfjtur wrote: post #22
easygenes wrote: post #7

This is very light and approachable but stops short of building the statistical intuition you want here. They fixate on the smoothness of squared errors without connecting that to the gaussian noise model and establishing how that relates to the predictive power against natural sorts of data.

Yes, and it seems like it could’ve been written in-part by an LLM. But, the LLM could take your criticism, improve upon the original, and iterate that way until you feel that it has produced something close to an optimal textbook. The one thing missing is soul. I noticeably don’t feel like there was anyone behind this writing.

Ah, we’re resorting to ad machinum today. :)

Re: How linear regression works intuitively and how it leads to gradient descent

#33

Post by jampekka »

easygenes wrote: post #21
jampekka wrote: post #15

The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…

OLS is a straightforward way to introduce GD, and although an analytic solution exists it becomes memory and IO bound at sufficient scale, so GD is still a practical option.

Computationally OLS is taking the pseudoinverse of the system matrix, which for dense systems has a complexity of O(samples * parameters^2). For some GD implementations the complexity of a single step is probably O(samples * parameters), so there could be a asymptotic benefit, but it's hard to imagine a case where the benefit is even realized, let alone makes a practical difference.

And in any case nobody uses GD for regressions for statistical analysis purposes. In practice Newton-Raphson or other more complicated schemes (with a lot higher computation, memory and IO demands) with a lot nicer convergence properties are used.

Re: How linear regression works intuitively and how it leads to gradient descent

#34

Post by easygenes »

jampekka wrote: post #33
easygenes wrote: post #21

Earlier quoted context omitted.

OLS is a straightforward way to introduce GD, and although an analytic solution exists it becomes memory and IO bound at sufficient scale, so GD is still a practical option.

Computationally OLS is taking the pseudoinverse of the system matrix, which for dense systems has a complexity of O(samples * parameters^2). For some GD implementations the complexity of a single step is probably O(samples * parameters), so there could be a asymptotic benefit, but it's hard to imagine a case where the benefit is even realized, let alone makes a practical difference. And in any case nobody uses GD for…

Mini batch and streaming GD make the benefits obvious and trivial. Closed form OLS is unbeatable so long as samples * params^2 is comfortably sitting in memory. You often lose that as soon as your p approaches 10^5, which is common these days. Soon as you need distributed, streaming, or your data is too tall and or too wide then first order methods are the point of call.

Re: How linear regression works intuitively and how it leads to gradient descent

#35

Post by jbjbjbjb »

jampekka wrote: post #15

The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…

I’ve always felt that ML introductions completely butcher OLS. When I was taught it in stats we had to consider the Gauss-Markov conditions and interpret the coefficients, we would study the residuals. ML introductions just focus getting good predictions.

Re: How linear regression works intuitively and how it leads to gradient descent

#36

Post by jampekka »

easygenes wrote: post #34
jampekka wrote: post #33

Earlier quoted context omitted.

Computationally OLS is taking the pseudoinverse of the system matrix, which for dense systems has a complexity of O(samples * parameters^2). For some GD implementations the complexity of a single step is probably O(samples * parameters), so there could be a asymptotic benefit, but it's hard to imagine a case where the benefit is even realized, let alone makes a practical difference. And in any case nobody uses GD for…

Mini batch and streaming GD make the benefits obvious and trivial. Closed form OLS is unbeatable so long as samples * params^2 is comfortably sitting in memory. You often lose that as soon as your p approaches 10^5, which is common these days. Soon as you need distributed, streaming, or your data is too tall and or too wide then first order methods are the point of call.

With batching it becomes SGD. If you're OK with approximations, you have e.g. randomized, reduced rank and streaming SVDs. And these tend have a lot nicer approximation and convergence properties than SGD.

What are the common cases for 10^5 parameter OLS? Perhaps something like weather models could include such computations?

Re: How linear regression works intuitively and how it leads to gradient descent

#37

Post by soVeryTired »

jbjbjbjb wrote: post #35
jampekka wrote: post #15

The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…

I’ve always felt that ML introductions completely butcher OLS. When I was taught it in stats we had to consider the Gauss-Markov conditions and interpret the coefficients, we would study the residuals. ML introductions just focus getting good predictions.

IMO that's the fundamental difference between statistics and ML. The culture of stats is about fitting a model and interpreting the fit, while the culture of ML is to treat the model as a black box.

That's one of the reasons that multicollinearity is seen as a big deal by statisticians, but ML practitioners couldn't give a hoot.

Re: How linear regression works intuitively and how it leads to gradient descent

#38

Post by lupire »

soVeryTired wrote: post #37
jbjbjbjb wrote: post #35

Earlier quoted context omitted.

I’ve always felt that ML introductions completely butcher OLS. When I was taught it in stats we had to consider the Gauss-Markov conditions and interpret the coefficients, we would study the residuals. ML introductions just focus getting good predictions.

IMO that's the fundamental difference between statistics and ML. The culture of stats is about fitting a model and interpreting the fit, while the culture of ML is to treat the model as a black box. That's one of the reasons that multicollinearity is seen as a big deal by statisticians, but ML practitioners couldn't give a hoot.

You are describing the difference between academic mathematician statisticians and "applied/engineering/actuarial/business" people who use statistics. The "black box" culture goes back to before ML and before both computing Machines M and statistical Learning (iterative models)

Re: How linear regression works intuitively and how it leads to gradient descent

#39

Post by lupire »

c7b wrote: post #5

One interesting property of least squares regression is that the predictions are the conditional expectation (mean) of the target variable given the right-hand-side variables. So in the OP example, we're predicting the average price of houses of a given size. The notion of predicting the mean can be extended to other properties of the conditional distribution of the target variable, such as the median or other quanti…

How does an upcoming college student, or worse an already graduate, learn statistics like this, with depth of understanding of the meaning of the math, vs just plug an chugging cookbook formulas and "proving" theorems mechanically without the deep semantics?

Re: How linear regression works intuitively and how it leads to gradient descent

#40

Post by lupire »

uniqueuid wrote: post #4
reify wrote: post #3

All thats wrong with the modern world https://www.ibm.com/think/topics/linear-regression A proven way to scientifically and reliably predict the future Business and organizational leaders can make better decisions by using linear regression techniques. Organizations collect masses of data, and linear regression helps them use that data to better manage reality, instead of relying on experience and intuition. You can…

While I get your point, it doesn't carry too much weight, because you can (and we often read this) claim the opposite: Linear regression, for all its faults, forces you to be very selective about parameters that you believe to be meaningful, and offers trivial tools to validate the fit (i.e. even residuals, or posterior predictive simulations if you want to be fancy). ML and beyond, on the other hand, throws you in a…

My maxim of statistics is that applied statistics is the art of making decisions under uncertainty, but people treat it like the science of making certainty out of data.
Post reply on HN