Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

Post reply on HN
Print view View original

71–80 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#71

Post by JadeNB »

monkeyelite wrote: post #61
lupire wrote: post #39

Earlier quoted context omitted.

How does an upcoming college student, or worse an already graduate, learn statistics like this, with depth of understanding of the meaning of the math, vs just plug an chugging cookbook formulas and "proving" theorems mechanically without the deep semantics?

Dont take the “for engineers” version. > and "proving" theorems mechanically I think you’ve have a bad experience because writing a proof is explaining deep understanding.

> I think you’ve have a bad experience because writing a proof is explaining deep understanding.

I think your wording is the key—coming up with a proof is creating deep understanding, but writing a proof very much need not be explaining or creating deep understanding. Writing a proof can be done mechanically, by both instructor and student, and, if done so, neither demonstrates nor creates understanding.

(Also, in statistics more than in almost any other mathematically based subject, while the rigorous mathematical foundations are important, a complete theoretical understanding of those foundations need not shed any light on the actual practice of statistics.)

Re: How linear regression works intuitively and how it leads to gradient descent

#73

Post by jwilber »

jascha_eng wrote: post #25

The amount of em dashes in this make this look very AI written. Which doesn't make it a bad piece but makes me more carefully check every sentence for errors.

This tired take is in every thread now. The sort of behavior better served by a Reddit bot, and just as annoying.

Re: How linear regression works intuitively and how it leads to gradient descent

#75

Post by jampekka »

esafak wrote: post #69
orlp wrote: post #24

Earlier quoted context omitted.

This isn't true. In practice people don't use the analytical solution for efficient linear regression, they use stochastic methods. Square error is used because it is the maximum likelihood estimator under the assumption that observation noise is normally distributed, not because it is analytical.

...because stochastic methods are implicit regularizers, leading to solutions that generalize better. Let's spell it out for those that don't know. https://www.inference.vc/notes-on-the-origin-of-implicit-reg...

OLS is a convex optimization problem, so this doesn't really apply. And for statistical analysis you really don't want to add poorly understood artificial noise to the parameter estimates anyway.

Re: How linear regression works intuitively and how it leads to gradient descent

#77

Post by kyllo »

ogogmad wrote: post #41
lupire wrote: post #38

Earlier quoted context omitted.

You are describing the difference between academic mathematician statisticians and "applied/engineering/actuarial/business" people who use statistics. The "black box" culture goes back to before ML and before both computing Machines M and statistical Learning (iterative models)

I suspect that the "black box" philosophy for statistics/ML is actually bad if you don't have a quick way of verifying the predictions. For instance, using PCA as a "black box" is perfectly fine if you're using it to de-noise readings from a camera or other instrument, because a human being can quickly tell if the de-noising is working correctly or not. But if you're using PCA to make novel discoveries, where you don…

You make a good point, though the difference between ML and statistics isn't just about interpreting and validating the model. It's about the "novel discoveries" part aka Doing Science.

Statistical modeling is done primarily in service of scientific discovery--for the purpose of making an inference (population estimate from a sample) or a comparison to test a hypothesis derived from a theoretical causal model of a real-world process before viewing data. The parameters of a model are interpreted because they represent an estimate of a treatment effect of some intervention.

Methods like PCA can be part of that modeling process either way, but analyzing and fitting models to data to mine it for patterns without an a priori hypothesis is not science.

Re: How linear regression works intuitively and how it leads to gradient descent

#78

Post by tibbar »

Some important context missing from this post (IMO) is that the data set presented is probably not a very good fit for linear regression, or really most classical models: You can see that there's way more variance at one end of the dataset. So even if we find the best model for the data that looks great in our gradient-descent-like visualization, it might not have that much predictive power. One common trick to deal with data sets like this is to map the data to another space where the distribution is more even and then build a model in that space. Then you can make predictions for the original data set by taking the inverse mapping on the outputs of the model.

Re: How linear regression works intuitively and how it leads to gradient descent

#79

Post by kyllo »

xadhominemx wrote: post #57
jampekka wrote: post #52

Earlier quoted context omitted.

Central limit theorem tells in practice that gaussian distributions is can be expected to be quite common. And it makes the gaussian distribution a good first guess. Least squares gives the ML estimate for gaussian residuals. I don't find this very direct, and there being a rationale doesn't mean that rationale is what in reality drives the usage. I mention the relation to the gaussian distribution. Which part of the…

This part is incorrect: “ The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution” OLS is popular because it gives correct answers as a result of the CLT

In other words, as economists say, because OLS is provably the BLUE (Best Linear Unbiased Estimator) aka the Gauss-Markov Theorem.

Re: How linear regression works intuitively and how it leads to gradient descent

#80

Post by throwaway7783 »

In the same vein, Karpathy's video series "Neural Networks from zero to hero"[0] touches upon a lot of this and intuitions as well. One of the best introductory series (even if you ignore the neural net part of it) and brushes on gradients, differentiation and what it means intuitively.

[0] https://youtu.be/VMj-3S1tku0?si=jq1cCSn5si17KK1o

Post reply on HN