Performing Linear Regression Using Ruby • Hacker News Board

Re: Performing Linear Regression Using Ruby

#41

Post by beagle3 » Thu, Sep 13, 2012, 3:20 AM UTC

rohitarondekar wrote: post #39

beagle3 wrote: post #32

Earlier quoted context omitted.

All regularization work I'm aware of uses W=I (an identity matrix). Where did you find this zero origin matrix? Note that your W does not guarantee invertability - e.g., if your original (0,0) is already 0.

This was shown by Professor Andrew in the Coursera ML class that's happening right now. Given n features x1 to xn we introduce x0 feature which is always set to 1. During the Regularization lectures the professor said that we don't need to control (or regularize) the theta0 (the parameter for x0) because it doesn't make a difference. I believe this is the reason W(0,0) is set to 0. The lectures are a little light on…

Ok, that clears it up:

He doesn't need to set W(0,0) to 1 specifically because he sets x0 to 0 (which guarantees a non-zero value in the covariance matrix).

But the standard way to do L2 regularization (also known as "ridge regression") is to add a scaled identity matrix (the entire diagonal set to be nonzero)

Re: Performing Linear Regression Using Ruby

#42

Post by rohitarondekar » Thu, Sep 13, 2012, 4:14 AM UTC

beagle3 wrote: post #41

rohitarondekar wrote: post #39

Earlier quoted context omitted.

This was shown by Professor Andrew in the Coursera ML class that's happening right now. Given n features x1 to xn we introduce x0 feature which is always set to 1. During the Regularization lectures the professor said that we don't need to control (or regularize) the theta0 (the parameter for x0) because it doesn't make a difference. I believe this is the reason W(0,0) is set to 0. The lectures are a little light on…

Ok, that clears it up: He doesn't need to set W(0,0) to 1 specifically because he sets x0 to 0 (which guarantees a non-zero value in the covariance matrix). But the standard way to do L2 regularization (also known as "ridge regression") is to add a scaled identity matrix (the entire diagonal set to be nonzero)

You mean set x0 to 1, right?

People who do linear regression at work don't add a x0 feature? During the lecture the prof. only said that adding a x0=1 for all samples m, is by convention and helps simplify the computation. Unless I missed something during the lecture that's the only explanation that was given.

Re: Performing Linear Regression Using Ruby

#43

Post by ucsd_surfNerd » Thu, Sep 13, 2012, 4:46 AM UTC

cheald wrote: post #37

ucsd_surfNerd wrote: post #35

Earlier quoted context omitted.

I did look at GSL :). I decided to write it by hand because I wanted to help people understand the underlying math. I find that too many people use libraries without understand the math which can become problematic especially when performing statistical analysis. Thanks for pointing out GSL. I appreciate the feedback.

It's definitely good to understand the underlying math, and it's a good article on "this is how to translate a formula into Ruby code", but given that the article is positioned as "We needed to solve this problem, and this is how we solved it", it seems like it'd make more sense to focus on solving it with the least work and the best performance, which is why I mentioned GSL. It's great to see other people doing stat…

I appreciate the feedback, especially when it is constructive criticism. It helps me understand how other people were interpreting the post. Perhaps I got the positioning slightly wrong as I wanted it to be more about teaching the basic math and how to translate that into Ruby. I will try and get the positioning better next time.

While we do some stats in Ruby it definitely doesn't represent the entire "this is how we solved it". In fact we use a large amount of R and Java to solve all our statistics problems.

Re: Performing Linear Regression Using Ruby

#44

Post by beagle3 » Thu, Sep 13, 2012, 11:01 AM UTC

rohitarondekar wrote: post #42

beagle3 wrote: post #41

Earlier quoted context omitted.

Ok, that clears it up: He doesn't need to set W(0,0) to 1 specifically because he sets x0 to 0 (which guarantees a non-zero value in the covariance matrix). But the standard way to do L2 regularization (also known as "ridge regression") is to add a scaled identity matrix (the entire diagonal set to be nonzero)

You mean set x0 to 1, right? People who do linear regression at work don't add a x0 feature? During the lecture the prof. only said that adding a x0=1 for all samples m, is by convention and helps simplify the computation. Unless I missed something during the lecture that's the only explanation that was given.

Yes , I did, thanks.

> People who do linear regression at work don't add a x0 feature?

Sometimes they do that; sometimes the data already has a subset known to have sum 1 (e.g., if you binary variables that reflect "one of n choices" which must be set), and in this case adding x0=1 makes things worse (from a numerical perspective) for many algorithms.

Regardless, I've always seen regulation theory stated with lambda*identity matrices.