Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linear Regression

simonwardjones.co.uk

Post reply on HN
Print view View original

11–20 of 71 posts

Re: Linear Regression

#11

Post by em500 »

stdbrouw wrote: post #3

If you enjoy these kinds of explanations, "Data Science from Scratch" by Joel Grus explains many machine learning algorithms and has you implement simple versions of them in Python as you read along. It also covers linear regression and I wonder if that book is where the author got the idea for this series of blogposts. Kudos anyway. Something of a nitpick, but one thing that both Simon Ward-Jones and Joel Grus miss…

I don't think they've missed the fact that for linear regression there's an algebraic solution, but machine learners typically treat linear regression a simple special case, and as soon as want to go a bit beyond you have to go with an iterative numerical optimizer anyway so why bother with the special case solution.

My main criticism on the article would be that the nitty gritty section only makes sense to a reader that has already done a linear algebra / multivariate calculus course. In which case they've likely already covered least squares in greater depth (including the exact solutions) than this article. So I don't really see the purpose of the math section, except maybe to signal that the writer has a descent understanding of the algorithmic detail.

Re: Linear Regression

#12

Post by q3k »

quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

Yeah, the leap from cute emoji variables to partial derivatives really threw me off. The vectorisation was a final punch in the gut (“I hope you can convince yourself, assuming you know matrix multiplication”).

Re: Linear Regression

#13

Post by simonwardjones »

quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

haha - I love the use of the word brutal here! It is a brutal leap.

I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d3 visual where you can slide the gradient and see the total error change!

I could always add the line of best fit approach and more intuition for the cost function after I introduce training data (and before the brutal leap)?

Re: Linear Regression

#15

Post by ojnabieoot »

Two comments here, and I am sorry if they come across as mean:

1) I know this article isn't aimed at me, but I do truly hate the excessive emojis.

2) More substantively: not once does the word "statistics" enter here. There is not a single illustration of the idea behind linear regression, which is staggeringly simple: it is just finding the best linear fit. Anyone who looks at a 2D scatterplot can do an approximate linear regression with their brains by visualizing a line that sorta fits the data. While the gradient descent stuff is useful for extending a machine learning algorithm, introducing linear regression this way obscures the actual concept - and indeed, obscures what machine learning actually is.

In my experience there are a huge number of data scientists who are ignorant of statistics and don't understand their models and algorithms. Although this is unfair, it is my impression that the author himself does not understand linear regression, even if they are able to write formulas using vectors. This article does a disservice to learners.

Re: Linear Regression

#16

Post by BrandoElFollito »

This is one of the rare articles which applies linear regression to something which is linear by nature (I am assuming good faith from the authors when they use "intuitively" to mean mean "we know our model is linear because ")

Whenever I look at generic information (news, newspapers, generic articles), someone is taking a cloud of points and, bam!, draws a straight line through them. I actually reached to some authors to ask why they did that.

Answers (if any) were varied, I had I think one person who actually said that the data is expected to be linear. Many could not see the point, up to "if it looks like a line, then we put a line".

Re: Linear Regression

#18

Post by redroot »

Great read! I really enjoy that the article explains the problem in three different mediums, and you walk away with some understanding even if you can only access the first part. Reminds me of the Wired videos where a concept would be explained at 5 different levels https://www.youtube.com/watch?v=eRkgK4jfi6M

Re: Linear Regression

#19

Post by em500 »

simonwardjones wrote: post #13
quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

haha - I love the use of the word brutal here! It is a brutal leap. I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d…

As I mentioned in another comment, your solution derivation doesn't make sense for someone who hasn't already seen/done it before. And for them, the intro seems a bit too cutesy.

It seems to me that you're trying to condense a few weeks of intro linear algebra into a single blog post. In my experience that only works as a refresher, not for someone starting from zero.

Re: Linear Regression

#20

Post by olegious »

I consider myself above average intelligence, but my math skills are sorely lacking (didn't make it beyond pre-calculus). The first section was simple to follow but I was completely lost as soon as he hit the "Nitty Gritty."

Any advice on what I need to learn to even begin to understand what's written in section 2?

Post reply on HN