Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

Post reply on HN
Print view View original

41–50 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#41

Post by ogogmad »

lupire wrote: post #38
soVeryTired wrote: post #37

Earlier quoted context omitted.

IMO that's the fundamental difference between statistics and ML. The culture of stats is about fitting a model and interpreting the fit, while the culture of ML is to treat the model as a black box. That's one of the reasons that multicollinearity is seen as a big deal by statisticians, but ML practitioners couldn't give a hoot.

You are describing the difference between academic mathematician statisticians and "applied/engineering/actuarial/business" people who use statistics. The "black box" culture goes back to before ML and before both computing Machines M and statistical Learning (iterative models)

I suspect that the "black box" philosophy for statistics/ML is actually bad if you don't have a quick way of verifying the predictions. For instance, using PCA as a "black box" is perfectly fine if you're using it to de-noise readings from a camera or other instrument, because a human being can quickly tell if the de-noising is working correctly or not. But if you're using PCA to make novel discoveries, where you don't have an independent way of checking those discoveries, then it might be outright essential to have a deep definition-theorem-proof style understanding of PCA. What do people think of this hunch?

The point about PCA applies to population genetics and psychometrics (IQ). Some conclusions have been derived using PCA that appear to be supported by little else, and these have come under question.

Re: How linear regression works intuitively and how it leads to gradient descent

#42

Post by BlueUmarell »

easygenes wrote: post #31
BlueUmarell wrote: post #16

Earlier quoted context omitted.

Any resource/link you know of that further develops your point?

CMU lecture notes [0] I think approach it in an intuitive way, starting from the Gaussian noise linear model, deriving log-likelihood, and presenting the analytic approach. Misses the bridge to gradient methods though. For gradients, Stanford CS229 [1] jumps right into it. [0] https://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/06/lectu... [1] https://cs229.stanford.edu/lectures-spring2022/main_notes.pd...

Thanks! will have a look..

Re: How linear regression works intuitively and how it leads to gradient descent

#43

Post by revskill »

stared wrote: post #28

I really recommend this explorable explanation: https://setosa.io/ev/ordinary-least-squares-regression/ And for actual gradient descent code, here is an older example of mine in PyTorch: https://github.com/stared/thinking-in-tensors-writing-in-pyt...

Google search is evil by not giving me those resources.

Re: How linear regression works intuitively and how it leads to gradient descent

#44

Post by jovial_cavalier »

akst wrote: post #23
easygenes wrote: post #7

This is very light and approachable but stops short of building the statistical intuition you want here. They fixate on the smoothness of squared errors without connecting that to the gaussian noise model and establishing how that relates to the predictive power against natural sorts of data.

It isn't too hard to find resources on this for anyone genuinely looking to get a deeper understanding of a topic. I think a blog post (likely written for SEO purposes, which is in no way a knock against the content) is probably the wrong place that kind of enlightenment, but I also think there are limits to the level of detail you can reasonable expect from a high level blog post. And for introductory content there'…

>I think a blog post... is probably the wrong place that kind of enlightenment

There's this site full of cool knowledgeable people called Hacker News which usually curates good articles with deep intuition about stuff like that. I haven't been there in years, though.

Re: How linear regression works intuitively and how it leads to gradient descent

#45

Post by sorcerer-mar »

revskill wrote: post #43
stared wrote: post #28

I really recommend this explorable explanation: https://setosa.io/ev/ordinary-least-squares-regression/ And for actual gradient descent code, here is an older example of mine in PyTorch: https://github.com/stared/thinking-in-tensors-writing-in-pyt...

Google search is evil by not giving me those resources.

This is an all-time great blog post for this line alone: "That's why we have statistics: to make us unsure about things."

The interactive visualizations are a great bonus though!

Re: How linear regression works intuitively and how it leads to gradient descent

#46

Post by xadhominemx »

jampekka wrote: post #15

The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…

That is incorrect. Least squares follows directly from the central limit theorem.

Re: How linear regression works intuitively and how it leads to gradient descent

#47

Post by ayhanfuat »

lupire wrote: post #39
c7b wrote: post #5

One interesting property of least squares regression is that the predictions are the conditional expectation (mean) of the target variable given the right-hand-side variables. So in the OP example, we're predicting the average price of houses of a given size. The notion of predicting the mean can be extended to other properties of the conditional distribution of the target variable, such as the median or other quanti…

How does an upcoming college student, or worse an already graduate, learn statistics like this, with depth of understanding of the meaning of the math, vs just plug an chugging cookbook formulas and "proving" theorems mechanically without the deep semantics?

Statistical Rethinking is quite good in explaining this stuff. https://xcelab.net/rm/

Re: How linear regression works intuitively and how it leads to gradient descent

#48

Post by xadhominemx »

jampekka wrote: post #30
orlp wrote: post #24

Earlier quoted context omitted.

This isn't true. In practice people don't use the analytical solution for efficient linear regression, they use stochastic methods. Square error is used because it is the maximum likelihood estimator under the assumption that observation noise is normally distributed, not because it is analytical.

If by stochastic methods you mean something like MCMC, they are increasing in popularity, but still used a lot less than analytical or numerical methods. And almost exclusively only for more complicated models than basic linear regression. Sampling methods have major downsides, and approximation methods like ADVI are becoming more popular. Though sampling vs approximations is a bit off topic, as neither usually have…

It’s not because of analytical convenience, it’s because of the central limit theorem.

Re: How linear regression works intuitively and how it leads to gradient descent

#49

Post by eitally »

FitCodIa wrote: post #19
alexey-salmin wrote: post #12

Earlier quoted context omitted.

That's the point, yes. "Piling up more and more data then stirring it until it works" stopped being a joke and turned out to be a practical approach. This can be seen as another occurence of the "bitter lesson": http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Thanks for the link to The Bitter Lesson. I indeed find the lesson that it describes unbearably bitter. Searching and learning, as used by the article, may discover patterns and results (due to infinite scaling of computation) that we, humans, are physically uncapable of discovering -- however, all those learnings will have no meaning , they will not expose any causality. This is what I find unbearable, as it implies…

Arguably, "the way our reasoning works" is probably a normal distribution but with a broad curve (and for some things, possibly a bimodal distribution), so trying to understand "why" is a fool's errand. It's more valuable to understand the input variables and then be able to calculate the likely output behaviors with error bars than to try to reduce the problem to a guaranteed if(this), then(that) equation. I don't particularly care why a person behaves a certain way in many cases, as long as 1) their behavior is generally within an expected range, and 2) doesn't harm themselves or others, and I don't see why I'd care any more about the behavior of an AI-driven system. As with most things, Safety first!

Re: How linear regression works intuitively and how it leads to gradient descent

#50

Post by mhb »

revskill wrote: post #43
stared wrote: post #28

I really recommend this explorable explanation: https://setosa.io/ev/ordinary-least-squares-regression/ And for actual gradient descent code, here is an older example of mine in PyTorch: https://github.com/stared/thinking-in-tensors-writing-in-pyt...

Google search is evil by not giving me those resources.

Kagi FTW?
Post reply on HN