> Regression is any algorithm that takes a collection of inputs and predicts an output.
I get that the author is writing for a certain audience, but this is a gross over-simplification.
Linear Regression
21–30 of 71 posts
Re: Linear Regression
#22I would prefer an introduction via the Moore-Penrose Pseudo-Inverse. It's a lot easier, imho
Introducing linear regression with gradient descent can also be really confusing for a newcomer. Gradient descent solves a problem (nonlinear optimization) which does not exist here.
Re: Linear Regression
#23If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway. Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss…
I’m working through Joel’s book and implementing the solutions in Swift:
https://github.com/melling/data-science-from-scratch-swift
Of the last 5, only the Linear Regression chapter is finished:
https://github.com/melling/data-science-from-scratch-swift/b...
Re: Linear Regression
#24> Regression is any algorithm that takes a collection of inputs and predicts an output. I get that the author is writing for a certain audience, but this is a gross over-simplification.
Depends on the tradition you are coming from. In machine learning it really is just that. A synonym for continuous-valued function approximation based on training data.
You may say it's only regression if it fits your favorite framework, like giving confidence values or goodness of fit etc, but that's not true in general. There so many variants, like Bayesian probabilistic regression, nonparametrics, neural nets, random forests etc.
Re: Linear Regression
#25I consider myself above average intelligence, but my math skills are sorely lacking (didn't make it beyond pre-calculus). The first section was simple to follow but I was completely lost as soon as he hit the "Nitty Gritty." Any advice on what I need to learn to even begin to understand what's written in section 2?
Machine learning math is basically calculus, linear algebra and probability theory. Pick up intro textbooks on these. Enjoying the dopamine rush of insight from blog posts can be more harmful than useful if overdone. You have to sit down and digest these things without diatractions, and that's not easy for most people outside a school environment. Consciously understand this fact and take an adult, planned approach to keep yourself on track if you truly want to understand things.
Re: Linear Regression
#26Great article ! But I must admit, I didn't expect it to be using gradient descent. I like the Wikipedia derivation as well [1]. It says that the minimization problem is convex, so you'll find a global minimum with gradient descent. But more interestingly, at the cost of inverting a potentially huge matrix, the article presents an elegant analytical solution to the problem.
[1] https://en.wikipedia.org/wiki/Ordinary_least_squares#Matrix/...
Re: Linear Regression
#27This is one of the most condescending things I've read in years
Re: Linear Regression
#28The next level:
Re: Linear Regression
#29I consider myself above average intelligence, but my math skills are sorely lacking (didn't make it beyond pre-calculus). The first section was simple to follow but I was completely lost as soon as he hit the "Nitty Gritty." Any advice on what I need to learn to even begin to understand what's written in section 2?
I mentioned Coursera in another comment but I think it's exactly what you are looking for.
The intro video for linear regression of one variable is free (I think the whole course is free) and gives a good taste of what's in the course:
https://www.coursera.org/lecture/machine-learning/model-repr...
Re: Linear Regression
#30This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…
haha - I love the use of the word brutal here! It is a brutal leap. I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d…
I'm very much not a fan of this introduction.
You use gradient descent, but do not introduce the normal equations. This is problematic for at least two reasons.
Case 1: Design matrix has full rank
Omitting the normal equations obfuscates what is really going on. You are inverting a matrix to solve the first order condition of a strictly convex objective, which therefore has a unique optimum.
Case 2: Design matrix does not have full rank
Omitting the normal equations hides the fact that there are multiple solutions to the first order condition. Gradient descent will find one of them, but you need a principled method for selecting among them. The Moore-Penrose pseudo-inverse method gives you the solution with the smallest L2 norm.
Omitting these details is setting learners up for failure.