Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Linear Regression

simonwardjones.co.uk

Post reply on HN
Print view View original

51–60 of 71 posts

Re: Linear Regression

#51

Post by edge17 »

simonwardjones wrote: post #13
quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

haha - I love the use of the word brutal here! It is a brutal leap. I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d…

Well, since everyone seems to be saying they're not a fan, I'll just add that I think it's great. Then again, I'm familiar with linear algebra, error functions, etc so maybe the writer is writing for a specific type of reader with a specific background. In spite of being familiar with the material, I still quite enjoy reading how other people explain these things.

Re: Linear Regression

#52

Post by quietbritishjim »

pps43 wrote: post #42
quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

> that's really the most obvious cost function To me Deming regression is the most obvious. It actually took me a long time to realize that x~y and y~x are in most cases different lines.

Oops, you're absolutely right. I remember back at secondary school a teacher going through exactly the process I described, with him asking us what the best choice of line would be. The first choice someone (not me sadly) suggested was indeed this one.

For others that, like me, don't already know it (at least by name): the Deming cost function is the sum of perpendicular distances to the points from the line, as opposed to measuring only the vertical component.

Edit: Actually it looks like Deming regression is a bit more subtle and statistical than just sum of perpendicular distances. A treatment of linear regression from a statistical perspective of trying untangle some random noise is very worthy, but I'd save it for a second lesson, following a first lessons that's just an unmotivated "let's choose the line that's somehow closest to all the points".

Re: Linear Regression

#53

Post by j7ake »

Why go through all the steps of deriving gradient descent when there is an analytical formula to get the estimate of the parameters ? Maybe I guess to scale up in big data contexts? But the analytical solution in my opinion gives more insight to the problem than the gradient descent solution

Re: Linear Regression

#54

Post by EForEndeavour »

tluyben2 wrote: post #34
ojnabieoot wrote: post #15

Two comments here, and I am sorry if they come across as mean: 1) I know this article isn't aimed at me, but I do truly hate the excessive emojis. 2) More substantively: not once does the word "statistics" enter here. There is not a single illustration of the idea behind linear regression, which is staggeringly simple: it is just finding the best linear fit. Anyone who looks at a 2D scatterplot can do an approximate…

> I do truly hate the excessive emojis. I am more in the camp of actually always finding them excessive. Especially in more serious, and possibly, interesting posts.

Same here. They're the mental equivalent of speed bumps on a highway.

Re: Linear Regression

#55

Post by em500 »

quietbritishjim wrote: post #52
pps43 wrote: post #42

Earlier quoted context omitted.

> that's really the most obvious cost function To me Deming regression is the most obvious. It actually took me a long time to realize that x~y and y~x are in most cases different lines.

Oops, you're absolutely right. I remember back at secondary school a teacher going through exactly the process I described, with him asking us what the best choice of line would be. The first choice someone (not me sadly) suggested was indeed this one. For others that, like me, don't already know it (at least by name): the Deming cost function is the sum of perpendicular distances to the points from the line, as oppo…

Deming regressions have some serious drawbacks compared to OLS, most importantly that it's not scale/unit-invariant: if you rescale your (e.g. use house area in square metres instead of square feet to predict house prices) you get different results.

This may be less of an issue for the machine leaner who's only interested in point predictions, but it's a serious concern for us old fashioned statisticians who are more interested in inference.

Re: Linear Regression

#56

Post by autokad »

em500 wrote: post #55
quietbritishjim wrote: post #52

Earlier quoted context omitted.

Oops, you're absolutely right. I remember back at secondary school a teacher going through exactly the process I described, with him asking us what the best choice of line would be. The first choice someone (not me sadly) suggested was indeed this one. For others that, like me, don't already know it (at least by name): the Deming cost function is the sum of perpendicular distances to the points from the line, as oppo…

Deming regressions have some serious drawbacks compared to OLS, most importantly that it's not scale/unit-invariant: if you rescale your (e.g. use house area in square metres instead of square feet to predict house prices) you get different results. This may be less of an issue for the machine leaner who's only interested in point predictions, but it's a serious concern for us old fashioned statisticians who are more…

i wouldn't dare to say its a 'serious drawback'. as soon as you add ridge to regression (something very common for statisticians to do), it's also no longer scale/unit-invariant.

Edit: this is also not just true for ridge, but lasso as well

Re: Linear Regression

#57

Post by saeranv »

simonwardjones wrote: post #13
quietbritishjim wrote: post #9

This article had a brutal leap from being aimed at someone with barely any understanding of maths (complete with friendly emojis) to use of cost functions (without any explanation) and associated code. It's bit like the "how to draw an owl" meme. A good article on linear regression, in my opinion, would break it down into three steps: 1. Spend a bit of time looking at cost functions. In principle linear regression is…

haha - I love the use of the word brutal here! It is a brutal leap. I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d…

There's a couple of really good things in this intro that I wanted to point out.

1. I appreciate the rigour that went into the math notation. I need to see equations to start to make sense of something. And if you're a beginner, this at least provides an intuition for starting to grok these visuals.

2. It's visually very clear, and structured clearly. Sounds trivial, but this kind of visual organization helps a lot to break down complex topics.

3. Intro -> Math theory -> Python breakdown. The best way to teach math.

4. No memes. I feel like there's a huge temptation in articles for beginners to litter the post with moving gifs and memes which are distracting and annoying.

Re: Linear Regression

#58

Post by em500 »

autokad wrote: post #56
em500 wrote: post #55

Earlier quoted context omitted.

Deming regressions have some serious drawbacks compared to OLS, most importantly that it's not scale/unit-invariant: if you rescale your (e.g. use house area in square metres instead of square feet to predict house prices) you get different results. This may be less of an issue for the machine leaner who's only interested in point predictions, but it's a serious concern for us old fashioned statisticians who are more…

i wouldn't dare to say its a 'serious drawback'. as soon as you add ridge to regression (something very common for statisticians to do), it's also no longer scale/unit-invariant. Edit: this is also not just true for ridge, but lasso as well

Personally I consider the lack of scale-invariance one of the main drawbacks of most common regularizers too. Again, not a big deal if all you're after are y-hat, a bit more concerning if you're interested in beta-hat.

Re: Linear Regression

#59

Post by the_af »

grenoire wrote: post #38
tluyben2 wrote: post #34

Earlier quoted context omitted.

> I do truly hate the excessive emojis. I am more in the camp of actually always finding them excessive. Especially in more serious, and possibly, interesting posts.

I would have loved to believe that they are a fad and will pass away, but I just found myself using them in the same way that I used to use phpBB emoticons. I think they're never going to go away, but at least we can try to eradicate them from professional discourse and education.

The homebanking login page from my bank greets me with "Hello :) How can we help you today?" and it makes me cringe every time. This is a major bank by the way.

I do use smileys here on HN, on facebook, and chatting, to signal I'm being friendly and non-confrontational, to preempt and defuse situations which could escalate into anger if someone misreads the tone.

Re: Linear Regression

#60

Post by spekcular »

rmrfstar wrote: post #30
simonwardjones wrote: post #13

Earlier quoted context omitted.

haha - I love the use of the word brutal here! It is a brutal leap. I wanted to give the ideas in the simplest way with friendly emojis and then go onto the more complex derivation with the cost function etc. I haven't explained the idea of the cost function enough. I think your idea for stating with the cost function (as a concept - not a formula) and absolute error may have been nicer to be fair. I can see a nice d…

I'm very much not a fan of this introduction. You use gradient descent, but do not introduce the normal equations. This is problematic for at least two reasons. Case 1: Design matrix has full rank Omitting the normal equations obfuscates what is really going on. You are inverting a matrix to solve the first order condition of a strictly convex objective, which therefore has a unique optimum. Case 2: Design matrix doe…

I agree with this 100%.

You may amused to learn that "How would you program a solver for a system of linear equations?" was an informal interview question for a top machine learning PhD program, and applicants were not looked upon favorably if they mindlessly gave gradient descent as an answer.

Post reply on HN