The Unreasonable Effectiveness of Random Forests
1–10 of 35 posts
Re: The Unreasonable Effectiveness of Random Forests
#2This is so true. Random forest should be the default choice for most problem sets.
Now we have The Unreasonable Effectiveness of Random Forests and The Unreasonable Effectiveness of Recurrent Neural Networks. We just need The Unreasonable Effectiveness of XGBoost (for winning Kaggle competitions) and we'll have the whole set.
Re: The Unreasonable Effectiveness of Random Forests
#3The article mentions that "The main drawback of Random Forests is the model size. You could easily end up with a forest that takes hundreds of megabytes of memory and is slow to evaluate."
A cool trick for speeding up trained random forests/gradient boosted decision trees is to dump the tree to C/ASM, compile it, and dlopen it as a function pointer. Depending on the model and the architecture, you can get an 8x speedup relative to an optimized C implementation that walks the tree.
There's an implementation for scikit-learn with benchmarks at https://github.com/ajtulloch/sklearn-compiledtrees if you're interested in this technique.
Re: The Unreasonable Effectiveness of Random Forests
#4The article mentions that "The main drawback of Random Forests is the model size. You could easily end up with a forest that takes hundreds of megabytes of memory and is slow to evaluate." A cool trick for speeding up trained random forests/gradient boosted decision trees is to dump the tree to C/ASM, compile it, and dlopen it as a function pointer. Depending on the model and the architecture, you can get an 8x speed…
(I) Indeed, it is a pretty good trick, and is simple to implement, given that the algorithm for evaluating a random forest is simple. Good on you for implementing it nicely for scikit-learn, benchmarking it, and making the code available.
I did something similar a few years ago when using a random forest as a heuristic to speed-up some minimax game tree search code. I don't think I still have the python scripts used to generate the compiled code (no great loss, they would have been terrible throwaway code) but amusingly enough I still have an example of a compiled random forest:
(warning: link #2 is a ~44k line cpp file of gotos encoding 50 compiled decision trees)
1. https://raw.githubusercontent.com/fcostin/hangman_cpp/master/forest.h
2. https://raw.githubusercontent.com/fcostin/hangman_cpp/master/forest.cpp
(II) On a different note, the article states "Another point that some might find a concern is that random forest models are black boxes that are very hard to interpret.". Well, yes and no. Decision trees are pretty easy to interpret -- you can always pick out one of the decision trees and look at it.
If one is interested in learning more about random forests I'd recommend taking a look at these notes from Breiman and Cutler (the folks responsible for the algorithm!): https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...
E.g. they discuss "variable importance" measures, as featured in the original R version of randomForest. There are a bunch of tools built around random forests for extracting some insight beyond the raw predictions, to aid interpretation! Use them! https://www.stat.berkeley.edu/~breiman/RandomForests/cc_home...
(III) Final comment: the article doesn't mention the Bayesian interpretation of ensembling a bunch of statistical models together. That's another potentially insightful way of thinking about why the method might be effective, and isn't limited to decision trees.
Re: The Unreasonable Effectiveness of Random Forests
#5This is so true. Random forest should be the default choice for most problem sets. Now we have The Unreasonable Effectiveness of Random Forests and The Unreasonable Effectiveness of Recurrent Neural Networks . We just need The Unreasonable Effectiveness of XGBoost (for winning Kaggle competitions) and we'll have the whole set. [1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
Another one is The Unreasonable Effectiveness of Deep Learning [1] by LeCun.
[1] http://on-demand.gputechconf.com/gtc/2014/webinar/gtc-expres...
Re: The Unreasonable Effectiveness of Random Forests
#6This is so true. Random forest should be the default choice for most problem sets. Now we have The Unreasonable Effectiveness of Random Forests and The Unreasonable Effectiveness of Recurrent Neural Networks . We just need The Unreasonable Effectiveness of XGBoost (for winning Kaggle competitions) and we'll have the whole set. [1] http://karpathy.github.io/2015/05/21/rnn-effectiveness/
I love random forests but IMO the default should be extremely randomized forests. It's interesting that he advocates random forests as ExtraTrees (sklearns extremely randomized forest implementation) is also visible on his images.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.65....
Re: The Unreasonable Effectiveness of Random Forests
#7I share the awe of RF (especially as they look naive, but turn out to be extremely good for a wide range of problems), however, the main problem is that they are black boxes. Typically, it is easy to get good results, but almost no insight, or further pointers.
Many times I ended up using linear regression (with properly engineered variables), or something as simple, because it gave almost as good results as RF, but I could interpret the results, inputs, etc. (And may task was academic or business analytics, so CV score was not the only figure of merit.)
Re: The Unreasonable Effectiveness of Random Forests
#8The article mentions that "The main drawback of Random Forests is the model size. You could easily end up with a forest that takes hundreds of megabytes of memory and is slow to evaluate." A cool trick for speeding up trained random forests/gradient boosted decision trees is to dump the tree to C/ASM, compile it, and dlopen it as a function pointer. Depending on the model and the architecture, you can get an 8x speed…
Is this often done? Can you provide a reference explaining this further? Possibly specific to ML models.
Re: The Unreasonable Effectiveness of Random Forests
#9I really enjoyed the point made on the first paragraph, especially the following phrase:
> Some like SVMs for the elegance of their formulation or the quality of the available implementations, some like decision rules for their simplicity and interpretability, and some are crazy about neural networks for their flexibility.
Nowadays it seems that ANN (and related algorithms) get all the credit. It's refreshing to see someone point out why some people actually prefer other methods.
Re: The Unreasonable Effectiveness of Random Forests
#10I wish there were some examples in the article of their effectiveness, mentions of best areas for their use, etc. Currently, as a non-ML expert, my takeaway from the article was "there's a thing called a Random Forest. That's quite cool."
Maybe I'm not the target audience, though.