I don't have anything useful to say, but, how the hell is that a "12 min read"?
I always find those counters to greatly overestimate reading speed, but for a technical article like this it's outright insulting, to be honest.
How linear regression works intuitively and how it leads to gradient descent
51–60 of 107 posts
Re: How linear regression works intuitively and how it leads to gradient descent
#52The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution. Makes it a bit weird example for gradient descent. There are plenty of error formulations that give a smooth loss function, and many even a convex one, but most don't have analytical solutions so they are solved via numerical optimization like GD. The main message is IMHO correct though: square…
That is incorrect. Least squares follows directly from the central limit theorem.
Central limit theorem tells in practice that gaussian distributions is can be expected to be quite common. And it makes the gaussian distribution a good first guess. Least squares gives the ML estimate for gaussian residuals. I don't find this very direct, and there being a rationale doesn't mean that rationale is what in reality drives the usage.
I mention the relation to the gaussian distribution. Which part of the comment is incorrect?
Re: How linear regression works intuitively and how it leads to gradient descent
#53Earlier quoted context omitted.
If by stochastic methods you mean something like MCMC, they are increasing in popularity, but still used a lot less than analytical or numerical methods. And almost exclusively only for more complicated models than basic linear regression. Sampling methods have major downsides, and approximation methods like ADVI are becoming more popular. Though sampling vs approximations is a bit off topic, as neither usually have…
It’s not because of analytical convenience, it’s because of the central limit theorem.
Not everything is a linear combination of large number of (IID) samples, and thus not everything is gaussian distributed.
Re: How linear regression works intuitively and how it leads to gradient descent
#54Earlier quoted context omitted.
It’s not because of analytical convenience, it’s because of the central limit theorem.
Not everything is a linear combination of large number of (IID) samples, and thus not everything is gaussian distributed.
You’re implying that many things are though.
Re: How linear regression works intuitively and how it leads to gradient descent
#55Earlier quoted context omitted.
While I get your point, it doesn't carry too much weight, because you can (and we often read this) claim the opposite: Linear regression, for all its faults, forces you to be very selective about parameters that you believe to be meaningful, and offers trivial tools to validate the fit (i.e. even residuals, or posterior predictive simulations if you want to be fancy). ML and beyond, on the other hand, throws you in a…
My maxim of statistics is that applied statistics is the art of making decisions under uncertainty, but people treat it like the science of making certainty out of data.
That sums it up exceptionally well.
Re: How linear regression works intuitively and how it leads to gradient descent
#56Re: How linear regression works intuitively and how it leads to gradient descent
#57Earlier quoted context omitted.
That is incorrect. Least squares follows directly from the central limit theorem.
Central limit theorem tells in practice that gaussian distributions is can be expected to be quite common. And it makes the gaussian distribution a good first guess. Least squares gives the ML estimate for gaussian residuals. I don't find this very direct, and there being a rationale doesn't mean that rationale is what in reality drives the usage. I mention the relation to the gaussian distribution. Which part of the…
This part is incorrect: “ The main practical reason why square error is minimized in ordinary linear regression is that it has an analytical solution”
OLS is popular because it gives correct answers as a result of the CLT
Re: How linear regression works intuitively and how it leads to gradient descent
#58Earlier quoted context omitted.
You’re implying that many things are though.
Yes, and I was explicit about it in another comment in this post.
Ok, so we all agree that in most cases the reason to minimize square error is that it’s the appropriate thing to minimize - not that it has an analytical solution, convenience or tradition.
Re: How linear regression works intuitively and how it leads to gradient descent
#59I really recommend this explorable explanation: https://setosa.io/ev/ordinary-least-squares-regression/ And for actual gradient descent code, here is an older example of mine in PyTorch: https://github.com/stared/thinking-in-tensors-writing-in-pyt...
Google search is evil by not giving me those resources.
Yeah - I wanted to post it here, but after searching for "linear regression explorable explanation" I got some other random links. Thankfully, I saved the PyTorch materials + https://pinboard.in/u:pmigdal/t:explorable-explanation.
Re: How linear regression works intuitively and how it leads to gradient descent
#60I really recommend this explorable explanation: https://setosa.io/ev/ordinary-least-squares-regression/ And for actual gradient descent code, here is an older example of mine in PyTorch: https://github.com/stared/thinking-in-tensors-writing-in-pyt...
Google search is evil by not giving me those resources.
Google does however provide this very nice course that explains these things in more detail: https://developers.google.com/machine-learning/crash-course