Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

11–20 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#11

Earlier quoted context omitted.

While I get your point, it doesn't carry too much weight, because you can (and we often read this) claim the opposite: Linear regression, for all its faults, forces you to be very selective about parameters that you believe to be meaningful, and offers trivial tools to validate the fit (i.e. even residuals, or posterior predictive simulations if you want to be fancy). ML and beyond, on the other hand, throws you in a…

That particular xkcd was funny until the LLMs came around

Well I'd say that prompt engineering is still exactly this?

Re: How linear regression works intuitively and how it leads to gradient descent

#12
post #9

Earlier quoted context omitted.

That particular xkcd was funny until the LLMs came around

Aren't LLMs also a pile of linear algebra?

That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach.

This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Re: How linear regression works intuitively and how it leads to gradient descent

#14
post #5

One interesting property of least squares regression is that the predictions are the conditional expectation (mean) of the target variable given the right-hand-side variables. So in the OP example, we're predicting the average price of houses of a given size. The notion of predicting the mean can be extended to other properties of the conditional distribution of the target variable, such as the median or other quanti…

[deleted]

Re: How linear regression works intuitively and how it leads to gradient descent

#15
The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent.

There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD.

The main message is IMHO correct though: square error (and its implicit gaussian noise assumption) is all too often used just per convenience and tradition.

Re: How linear regression works intuitively and how it leads to gradient descent

#16

This is very light and approachable but stops short of building the statistical intuition you want here. They fixate on the smoothness of squared errors without connecting that to the gaussian noise model and establishing how that relates to the predictive power against natural sorts of data.

Any resource/link you know of that further develops your point?

Re: How linear regression works intuitively and how it leads to gradient descent

#17
post #9

Earlier quoted context omitted.

Aren't LLMs also a pile of linear algebra?

That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach. This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Uh, it's funny because it works. It came out at a point where that approach was already being used in plenty of applications.

Re: How linear regression works intuitively and how it leads to gradient descent

#19
post #9

Earlier quoted context omitted.

Aren't LLMs also a pile of linear algebra?

That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach. This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Thanks for the link to The Bitter Lesson.

I indeed find the lesson that it describes unbearably bitter. Searching and learning, as used by the article, may discover patterns and results (due to infinite scaling of computation) that we, humans, are physically uncapable of discovering -- however, all those learnings will have no meaning, they will not expose any causality. This is what I find unbearable, as it implies that the real world must ultimately remain impervious to human cognizance; it implies that our meaning- and causality-based human reasoning ultimately falls short to model the world, while general, computation-only methods (given ever-growing computing power) at least "converges" to a faithful (but meaningless) description of the world.

See examples like protein folding, medicine research, AI-assisted diagnosis, self driving cars. We're going to rely on their results, but we'll never know why those results work. We're not going to reject self-driving cars if those cars save lives per same distance driven and/or same time driven; however, we're going to sit in, and drive, those cars blind. To me, that's an unbearable thought, even apart from the possibility that at some point the system might break down, and cause a huge accident inexplicably. An inexplicable misbehavior of the system is of course catastrophic, but to me, even the inexplicable proper behavior of the system is an unsettling thought -- because it is inexplicable.

Edited to add: I think the phrase "how we think we think" is awesome in the essay. We don't even know how our reasoning works, so trying to "machinize" those misconceptions is likely bound to fail.

Re: How linear regression works intuitively and how it leads to gradient descent

#20

Speaking of linear regression, can any of you recommend an online course or book that deep dives into fitting linear models?

Most intro to stats courses will do

I did the Stats I -> II -> II pipeline at uni but you should be fitting basic linear models by the end of Stats I

Post reply on HN