Linear Regression
simonwardjones.co.uk
Linear Regression
1–10 of 71 posts
Re: Linear Regression
#2Explanation, maths and code
Re: Linear Regression
#3If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway.
Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss is that linear regression is typically not implemented using gradient descent at all, there's an analytical solution and you can get the beta coefficients with straightforward matrix algebra. It's much harder to explain than gradient descent so I get why they don't bother, but on the other hand without that background it's hard to see why everybody talks about linear regression all the time when with gradient descent or any other numerical optimizer there's really no limit to what f(x) can look like.
Re: Linear Regression
#4This is a very good explanation! (And a great blog btw). I especially like the math symbols explanation. This is something I always have a hard time understanding/remembering. (It’s very hard to lookup math symbols om Google :P) and great that it is followed by a Python example!
Re: Linear Regression
#5If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway. Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss…
Another resource I'd recommend is Andrew Ng's Machine Learning course on Coursera.
Re: Linear Regression
#6Explanation, maths and code
Well done. I look forward to digging into this more this weekend.
Re: Linear Regression
#7I would prefer an introduction via the Moore-Penrose Pseudo-Inverse. It's a lot easier, imho
Re: Linear Regression
#8If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway. Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss…
> it's hard to see why everybody talks about linear regression all the time when with gradient descent or any other numerical optimizer there's really no limit to what f(x) can look like.
Even though there is no limit to what f can look like (except being differentiable), the kind of models that you can reasonably fit and obtain good performance from is still very much limited by how much data you have, and of which quality. And data is indeed the bottleneck in the vast majority of cases.
Re: Linear Regression
#9This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme.
A good article on linear regression, in my opinion, would break it down into three steps:
1. Spend a bit of time looking at cost functions. In principle linear regression is finding the "best" line, where by "best" we consider all possible lines (yes, all uncountably infinite of them), compute the cost function for each one, and pick the one where the cost comes out lowest. You want to show a few example lines on top of some example points and label their costs. Start with absolute deviation (i.e. l1 norm) to start with - let's face it, that's really the most obvious cost function if you don't already know what comes next - then contrast with least squares (i.e. l2 norm). For example, note that least squares cost function "cares" more about points that are a particularly long way away.
2. Admit that, OK, we do want an algorithm more sensible than "try every possible line, labourously computing the cost of each one". Now you can talk about gradient descent - and I mean WHY you use gradient descent, not the computation. And now you can mention that least squares is differentiable, so solves nicely for gradient descent, which is the real reason we tend to prefer it over absolute deviation.
3. Finally, after both of those you can solve the gradient descent equations and show some associated code.
Re: Linear Regression
#10If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway. Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss…
I had never actually seen "Data Science from Scratch" - sounds like a book I should read!
Fair point about the gradient descent Vs normal equations closed form solution. I am planning on working through a few algorithms so thought it would be better to introduce gradient descent with something simple before talking about gradient boosted decision trees and Neural Networks. Also I would have to explain more complex matrix stuff like invertibility issues and linear dependance like you said.
I guess I just dodged that bullet and went for gradient descent. Maybe another post for the linear algebra fans! Thanks for reading though!