B = (X^TX)^(-1)X^TY anyone?
Put a pseudoinverse in there :)
[deleted]
Performing Linear Regression Using Ruby
21–30 of 44 posts
Re: Performing Linear Regression Using Ruby
#22B = (X^TX)^(-1)X^TY anyone?
Implementation tip - you don't need to invert that matrix! [1]. Whenever you see an equation of the form Ab = c and you want to find b, you should use a function equivalent to lu_solve (in the GNU Scientific Library) or the left-divide operator in Octave/Matlab (b = A\c), which will use the LU factorization algorithm [2] without ever computing A^(-1).
This is normally faster, more numerically stable and more space efficient. Even better, once you've computed the LU factorization of A once (which takes O(n^3) operations) you can then solve Ab = c for many different values of b and c in O(n^2) operations, by caching the factorization of A.
[1] http://www.johndcook.com/blog/2010/01/19/dont-invert-that-ma... [2] http://en.wikipedia.org/wiki/LU_decomposition
Re: Performing Linear Regression Using Ruby
#23B = (X^TX)^(-1)X^TY anyone?
Implementation tip - you don't need to invert that matrix! [1]. Whenever you see an equation of the form Ab = c and you want to find b, you should use a function equivalent to lu_solve (in the GNU Scientific Library) or the left-divide operator in Octave/Matlab (b = A\c), which will use the LU factorization algorithm [2] without ever computing A^(-1). This is normally faster, more numerically stable and more space ef…
[deleted]
Re: Performing Linear Regression Using Ruby
#24I think it is especially important to note that linear regression assumes that the relationship between the variables is, well, linear , and that in the real world, it very rarely actually is. At best, a big asterisk should come from any of these results if you didn't have someone with actual experience validate your design/proposed analyses first.
You can use the same techniques used in Linear Regression (with multiple features) to do Polynomial Regression. For example suppose you have two features x1 and x2, you can add higher order features like x1x2 or x1^2 or x2^2 or a combination of these. While doing linear regression you treat these terms as individual features so x1x2 is a feature say x3. This way you can fit non-linear data with a non-linear curve. However there is a problem of overfitting, i.e your curve may try to be too greedy and fit the data perfectly, but that's not what you want. So Regularization is used to lower the contributions of the higher order terms.
Wikipedia has an article on Polynomial Regression: http://en.wikipedia.org/wiki/Polynomial_regression
P.S I'm doing this course https://www.coursera.org/course/ml so my knowledge may not be entirely correct so take everything I've said with a pinch of salt. :)
Re: Performing Linear Regression Using Ruby
#25Earlier quoted context omitted.
Python is surprisingly heavy-duty. But my kingdom for a seamlessly distributed or parallelized version of NumPy/SciPy! How nice would it be to just enter "C = A * B", with A living as a sparse CSC across many nodes?
Would Disco ( http://discoproject.org/ ) work for you?
No, the sparse matrix code in SciPy is plain C (not even multi-core, let alone distributed).
EDIT: or did you mean Disco offers distributed sparse CSC operations?
Re: Performing Linear Regression Using Ruby
#26B = (X^TX)^(-1)X^TY anyone?
Implementation tip - you don't need to invert that matrix! [1]. Whenever you see an equation of the form Ab = c and you want to find b, you should use a function equivalent to lu_solve (in the GNU Scientific Library) or the left-divide operator in Octave/Matlab (b = A\c), which will use the LU factorization algorithm [2] without ever computing A^(-1). This is normally faster, more numerically stable and more space ef…
According to Wikipedia[1]Matrix Inversion is also O(n^3) and according to the note can be even better by inverting smaller matrices within and multiplying[2].
My linear algebra isn't very good so I'll have to look into LU Factorization but is there vast difference between the computational performance of the two operations? (Assuming you don't need solve Ab=c for different values of b and c)
Also is LU Factorization used often in machine learning instead of inverse?
[1]: http://en.wikipedia.org/wiki/Computational_complexity_of_mat...
[2]: Because of the possibility to blockwise invert a matrix, where an inversion of an n×n matrix requires inversion of two half-sized matrices and 6 mulitplications between two half-sized matrices, and since matrix multiplication has a lower bound of Ω(n2 log n) operations[17], it can be shown that a divide and conquer algorithm that uses blockwise inversion to invert a matrix runs with the same time complexity as the matrix multiplication algorithm that is used internally. source is [1]
Re: Performing Linear Regression Using Ruby
#27Ruby is great for data prep, basic calculations, web app development, and scraping aggregating data and R for visualization. I find them to be a joyful combination. Great for small data sets, quick estimations, and various small projects. Larger data sets and performance intensive operations are better handled in Python (or Java, C++, etc). Lots of statistical analysis is way below the threshold of Hadoop and company…
I mostly agree with this, because I do like Ruby even though I don't use it. The missing link here, and the reason Python gets more love from the data community, is that Python scales down to the smaller data sets as well as it handles big ones. (Not sure if you ment it couldn't, but the distinction you make implies that.)
Agreed, Python scales down and so is also good for small tasks. What I was saying is that Ruby - as much as I like it - does not generally scale up beyond a certain point.
Both R and Ruby have had issues with large data sets which have been addressed to some degree in more different distributions and more recent releases. Python is ready out of the box for large data sets. So what I meant to communicate is that if you know that you are going to be dealing with a large data set, you might as well go straight to python.
Re: Performing Linear Regression Using Ruby
#28Earlier quoted context omitted.
Python is surprisingly heavy-duty. But my kingdom for a seamlessly distributed or parallelized version of NumPy/SciPy! How nice would it be to just enter "C = A * B", with A living as a sparse CSC across many nodes?
Would Disco ( http://discoproject.org/ ) work for you?
I don't think MR is a good abstraction for implementing linear algebra, and I expect the overhead to be too high (although I don't have numbers to back that up). For large problems (>> couple of machines worth of RAM), you use big iron HPC solutions, or you avoid 'exact' linear algebra altogether to focus on one-pass algorithms.
For example, instead of computing an exact SVD, you will use something like Hebbian algorithm to compute the SVD in a streaming manner (that's what Mahaout implements for example).
Re: Performing Linear Regression Using Ruby
#29...and at the end he shows you how to do the same thing with two lines of R. It's very useful to code basic statistical algorithms yourself so you understand how they work, but for any serious analysis you'll get more reliable and performant results with a library.
Totally agree. And if you want to call R from ruby to export the analytics to a web page for example, there are many options, such as this one: https://github.com/clbustos/Rserve-Ruby-client
Re: Performing Linear Regression Using Ruby
#30I think it is especially important to note that linear regression assumes that the relationship between the variables is, well, linear , and that in the real world, it very rarely actually is. At best, a big asterisk should come from any of these results if you didn't have someone with actual experience validate your design/proposed analyses first.
Someone once said that all models are wrong, but some are more useful than others (paraphrasing).