Would you suggest any books/resources to learn the theory behind these implementations so a newbie can follow along?
Machine Learning from scratch: Bare bones implementations in Python
21–30 of 67 posts
Re: Machine Learning from scratch: Bare bones implementations in Python
#22This is impressive, and kindof exactly what I am in the process of doing. It's certainly the best way to get familiar with the internal workings of these methods than just tune parameters like an oblivious albeit theoretically informed monkey. How long did it take you to do them?
Re: Machine Learning from scratch: Bare bones implementations in Python
#23One quick comment: in general it is a bad idea to compute the inverse of a matrix (to solve a linear system). It's much better to compute the QR factorization or SVD instead (or simply call least square solver). See for example: https://www.johndcook.com/blog/2010/01/19/dont-invert-that-m...
It's usually even better to use iterative methods
Re: Machine Learning from scratch: Bare bones implementations in Python
#24Would you suggest any books/resources to learn the theory behind these implementations so a newbie can follow along?
I can recommend Applied Predictive Modeling by Max Kuhn: http://appliedpredictivemodeling.com/
Re: Machine Learning from scratch: Bare bones implementations in Python
#25This is impressive, and kindof exactly what I am in the process of doing. It's certainly the best way to get familiar with the internal workings of these methods than just tune parameters like an oblivious albeit theoretically informed monkey. How long did it take you to do them?
It certainly is! Thanks. :) I have been working on it for about three weeks now.
Re: Machine Learning from scratch: Bare bones implementations in Python
#26One quick comment: in general it is a bad idea to compute the inverse of a matrix (to solve a linear system). It's much better to compute the QR factorization or SVD instead (or simply call least square solver). See for example: https://www.johndcook.com/blog/2010/01/19/dont-invert-that-m...
It's usually even better to use iterative methods
GMRES is definitely my go-to these days. Though it is worth noting that direct methods do have a benefit of letting you quickly solve many successive linear problems involving the same matrix, but different right-hand sides. But iterative methods scale very well for large sparse problems that they are very often the only tool to consider.
Block Krylov methods are a thing , but I haven't experimented with them yet.
Re: Machine Learning from scratch: Bare bones implementations in Python
#27Would you suggest any books/resources to learn the theory behind these implementations so a newbie can follow along?
Pattern recognition and machine learning by Bishop is one of the canonical text books. It helps to have a linear algebra background, it includes a refresher though
Re: Machine Learning from scratch: Bare bones implementations in Python
#28Earlier quoted context omitted.
It's usually even better to use iterative methods
GMRES is definitely my go-to these days. Though it is worth noting that direct methods do have a benefit of letting you quickly solve many successive linear problems involving the same matrix, but different right-hand sides. But iterative methods scale very well for large sparse problems that they are very often the only tool to consider. Block Krylov methods are a thing , but I haven't experimented with them yet.
For solving linear systems there are recent mind-blowing methods by R. M. Gower and P. Richtarik:
https://arxiv.org/abs/1506.03296
http://www.maths.ed.ac.uk/~richtarik/papers/SDA.pdf
Plus R. M. Gower is fantastically nice and enthusiastic so there's that
And thanks for the link !
Re: Machine Learning from scratch: Bare bones implementations in Python
#29Very cool! I have actually been planning to do exactly what you did, sir :)
Not sure what you had in mind, but a Python 3 version of this would be great!
Re: Machine Learning from scratch: Bare bones implementations in Python
#30This is impressive, and kindof exactly what I am in the process of doing. It's certainly the best way to get familiar with the internal workings of these methods than just tune parameters like an oblivious albeit theoretically informed monkey. How long did it take you to do them?
It certainly is! Thanks. :) I have been working on it for about three weeks now.
> I have been working on it for about three weeks now.
That's not a whole lot, you're quick