51–60 of 67 posts
This could become a fantastic resource for anybody who is teaching machine learning. One vital improvement suggestion to make that path attractive would be if the Jupyter notebook format were used. It would be easier to add more documentation and references. But in any case, thanks for sharing!
Would you suggest any books/resources to learn the theory behind these implementations so a newbie can follow along?
Introduction to statistical learning http://www-bcf.usc.edu/~gareth/ISL/
Earlier quoted context omitted.
It's usually even better to use iterative methods
GMRES is definitely my go-to these days. Though it is worth noting that direct methods do have a benefit of letting you quickly solve many successive linear problems involving the same matrix, but different right-hand sides. But iterative methods scale very well for large sparse problems that they are very often the only tool to consider. Block Krylov methods are a thing , but I haven't experimented with them yet.
Also, you can also use an approximate LU-decomp as a preconditioner for Krylov methods.
Similarly, Tykhonov regularization (solving for a range of slightly perturbed matrices "A + labda I" where labda is a parameter) are easily tackled using Krylov subspace methods by noting that the Krylov subspace is invariant under shifts like these. So only once an orthonormal basis must be found for the Krylov subspace, which can then be used for every lamda of interest.
Earlier quoted context omitted.
GMRES is definitely my go-to these days. Though it is worth noting that direct methods do have a benefit of letting you quickly solve many successive linear problems involving the same matrix, but different right-hand sides. But iterative methods scale very well for large sparse problems that they are very often the only tool to consider. Block Krylov methods are a thing , but I haven't experimented with them yet.
For solving linear systems there are recent mind-blowing methods by R. M. Gower and P. Richtarik: https://arxiv.org/abs/1506.03296 http://www.maths.ed.ac.uk/~richtarik/papers/SDA.pdf Plus R. M. Gower is fantastically nice and enthusiastic so there's that And thanks for the link !
If your goal is to solve say a LS problem, why not go for CGLS? http://web.stanford.edu/group/SOL/software/cgls/
One quick comment: in general it is a bad idea to compute the inverse of a matrix (to solve a linear system). It's much better to compute the QR factorization or SVD instead (or simply call least square solver). See for example: https://www.johndcook.com/blog/2010/01/19/dont-invert-that-m...
It's usually even better to use iterative methods
Direct methods as Gaussian elimination with pivoting are proven to be stable. Iterative methods are not but can be a lot cheaper in computational costs. Also they can stop when a certain relative residual is reached, unlike direct methods.
If iterative methods like Krylov subspace methods where stable, then they could actually be seen as direct methods themselves, as the Krylov subspace has at most a dimension of N, where N is the number of unknowns; so after at most N iterations the solution could be extracted exactly from the search space. In practice this is not the case due to rounding errors.
In your RandomForest implementation, on the line in fit() where you're building the training subsets to give to each tree, it appears that your bagging approach doesn't use 'sampling with replacement' strategy. idx = np.random.choice(range(n_features), size=self.max_features, replace=False) It would appear that the replace=False prevents the 'sampling with replacement' behavior usually implemented by bagging algorith…
Great resource, but it could be a phenomenal resource if you documented each method and explained how and why it does what it does. Don't get me wrong, having working code to play with is key, but when you don't fully grasp the concepts behind it, an explanation can become so valuable. That being said, you've included names, so research can be done. Great work and I hope you're enjoying it!
sci-kit learn is excellent, but their implementations are a bit to complicated to learn from. this is for people who don't just want to tune parameters but build the whole thing from scratch I can buy buy a pie all the fix-ins from a bakery, or I can buy the ingredients myself, and make it to exactly my liking. it may not be a professional.