Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How linear regression works intuitively and how it leads to gradient descent

briefer.cloud

Post reply on HN
Print view View original

61–70 of 107 posts

Re: How linear regression works intuitively and how it leads to gradient descent

#61

Post by monkeyelite »

lupire wrote: post #39
c7b wrote: post #5

One interesting property of least squares regression is that the predictions are the conditional expectation (mean) of the target variable given the right-hand-side variables. So in the OP example, we're predicting the average price of houses of a given size. The notion of predicting the mean can be extended to other properties of the conditional distribution of the target variable, such as the median or other quanti…

How does an upcoming college student, or worse an already graduate, learn statistics like this, with depth of understanding of the meaning of the math, vs just plug an chugging cookbook formulas and "proving" theorems mechanically without the deep semantics?

Dont take the “for engineers” version.

> and "proving" theorems mechanically

I think you’ve have a bad experience because writing a proof is explaining deep understanding.

Re: How linear regression works intuitively and how it leads to gradient descent

#62

Post by disgruntledphd2 »

ayhanfuat wrote: post #47
lupire wrote: post #39

Earlier quoted context omitted.

How does an upcoming college student, or worse an already graduate, learn statistics like this, with depth of understanding of the meaning of the math, vs just plug an chugging cookbook formulas and "proving" theorems mechanically without the deep semantics?

Statistical Rethinking is quite good in explaining this stuff. https://xcelab.net/rm/

Basically all of the Andrew Gelman books are also good.

Data Analysis... https://sites.stat.columbia.edu/gelman/arm/ Regression and Other Stories: https://avehtari.github.io/ROS-Examples/

Wasserman's All of Statistics is a really good introduction to mathematical statistics (the Gelman stuff above are more practically and analytically focused).

But yeah, it would probably be easier to find a good statistics course at a local university and try to audit it or do it at night.

Re: How linear regression works intuitively and how it leads to gradient descent

#64

Post by billbrown »

mhb wrote: post #50
revskill wrote: post #43

Earlier quoted context omitted.

Google search is evil by not giving me those resources.

Kagi FTW?

That was my initial thought, too. But I didn't know what the original Google search consisted of and the site didn't show up in a couple Kagi searches I tried. (Aside from the obvious titular one, of course.)

Re: How linear regression works intuitively and how it leads to gradient descent

#65

Post by rogue7 »

I built a small static web app [0] (with svelte and tensorflow js) that shows gradient descent. It has two kind of problems: wave (the default) and linear. In the first case, the algorithm learns y = ax + b ; in the second, y = cos(ax + b). The training data is generated from these functions with some noise.

I spent some time making it work with interpolation so that the transitions are smooth.

Then I expanded to another version, including a small neural network (nn) [1].

And finally, for the two functions that have a 2d parameter space, I included a viz of the loss [2]. You can click on the 2d space and get a new initial point for the descent, and see the trajectory.

Never really finished it, though I wrote a blog post about it [3]

[0] https://gradfront.pages.dev/

[1] https://f36dfeb7.gradfront.pages.dev/

[2] https://deploy-preview-1--gradient-descent.netlify.app/

[3] https://blog.horaceg.xyz/posts/need-for-speed/

Re: How linear regression works intuitively and how it leads to gradient descent

#66

Post by Workaccount2 »

dalmo3 wrote: post #51

I don't have anything useful to say, but, how the hell is that a "12 min read"? I always find those counters to greatly overestimate reading speed, but for a technical article like this it's outright insulting, to be honest.

It's the common trap of trying to teach, and why teaching is so much more difficult than it appears.

When you intimately understand a topic, you have an intuition that naturally paves over gaps and bumps. This is excellent for getting work done, but terrible for teaching. Your road from start to finish is 12 minutes, and without that knack for teaching, you are unable to see what that road looks like to a beginner.

Re: How linear regression works intuitively and how it leads to gradient descent

#67

Post by disgruntledphd2 »

xadhominemx wrote: post #57
jampekka wrote: post #52

Earlier quoted context omitted.

Central limit theorem tells in practice that gaussian distributions is can be expected to be quite common. And it makes the gaussian distribution a good first guess. Least squares gives the ML estimate for gaussian residuals. I don't find this very direct, and there being a rationale doesn't mean that rationale is what in reality drives the usage. I mention the relation to the gaussian distribution. Which part of the…

This part is incorrect: “ The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution” OLS is popular because it gives correct answers as a result of the CLT

And it has an analytical solution, which was important before computing (and still makes it quicker today).

Re: How linear regression works intuitively and how it leads to gradient descent

#68

Post by JadeNB »

rogue7 wrote: post #65

I built a small static web app [0] (with svelte and tensorflow js) that shows gradient descent. It has two kind of problems: wave (the default) and linear. In the first case, the algorithm learns y = ax + b ; in the second, y = cos(ax + b). The training data is generated from these functions with some noise. I spent some time making it work with interpolation so that the transitions are smooth. Then I expanded to ano…

> It has two kind of problems: wave (the default) and linear. In the first case, the algorithm learns y = ax + b ; in the second, y = cos(ax + b).

Are "first" and "second" switched here?

Re: How linear regression works intuitively and how it leads to gradient descent

#69

Post by esafak »

orlp wrote: post #24
jampekka wrote: post #15

The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…

This isn't true. In practice people don't use the analytical solution for efficient linear regression, they use stochastic methods. Square error is used because it is the maximum likelihood estimator under the assumption that observation noise is normally distributed, not because it is analytical.

...because stochastic methods are implicit regularizers, leading to solutions that generalize better. Let's spell it out for those that don't know.

https://www.inference.vc/notes-on-the-origin-of-implicit-reg...

Re: How linear regression works intuitively and how it leads to gradient descent

#70

Post by lucasfcosta »

liamwire wrote: post #29
jascha_eng wrote: post #25

The amount of em dashes in this make this look very AI written. Which doesn't make it a bad piece but makes me more carefully check every sentence for errors.

I know this is repeated ad nauseam by now, but as an ardent user of em dashes for many years pre-LLM, I think this a bad heuristic.

Co-author and founder of Briefer here.

I used to use em dashes before they were cool. I actually learned about them when I emailed a guy who's a software engineer at Genius and also writes for The New Yorker and The Atlantic.

I asked him for tips on how to write well and he recommended that I read Steven Pinker's "The Sense of Style", which uses em dashes exhaustively, and explains when and why one should use them.

It also pains me that I can't use them anymore or else people will think an AI did the writing.

Post reply on HN