The Unreasonable Effectiveness of Random Forests • Hacker News Board

Re: The Unreasonable Effectiveness of Random Forests

#31

Post by micro_cam » Sat, Sep 19, 2015, 8:08 PM UTC

thenomad wrote: post #10

I wish there were some examples in the article of their effectiveness, mentions of best areas for their use, etc. Currently, as a non-ML expert, my takeaway from the article was "there's a thing called a Random Forest. That's quite cool." Maybe I'm not the target audience, though.

RF's work well on heterogenous data (ie a mix of data types from difrent sources which might include numerical, categorical and text data etc. They also tend to be fairly resistance to noise in the data and to work well on data sets that are wide (ie have more features/dimensions then cases/observations). Finally they are fairly resistant to overfitting (ie fitting quirks in the training data instead of generalizable signal) and don't need too much parameter tuning.

This is largely because they are a randomized ensemble of weaker models. Individual decision trees are quite prone to overfitting and other issues but in an rf you grow a bunch of them on diffrent bootstrap samples of the data and let them vote and it turns out the combined performance is much better and much less error prone then a single model.

Specific examples where they work well include genetic data (many more noisy variables then observations) and customer/consumer data. They also get used in image data and signal processing but deep neural networks are recently tending to beat them here and in similar less heterogenous data sets.

Re: The Unreasonable Effectiveness of Random Forests

#32

Post by atpaino » Sat, Sep 19, 2015, 8:51 PM UTC

Nice post. However, I was kind of surprised to see the author list interpretability as one of the drawbacks of random forests:

> Another point that some might find a concern is that random forest models are black boxes that are very hard to interpret.

I generally agree that random forests are more difficult to interpret than linear models such as logistic regression, but I think they're still far more interpretable than more comparable non-linear models such as neural networks or SVMs with non-linear kernels. At the end of the day, a random forest is just a bunch of decision trees, each of which are very straightforward for humans to understand. Additionally, there are a number of straightforward methods available for assessing the importance of each individual feature in a random forest in aggregate ([1], section 15.4). Neural networks, on the other hand, result in models that are too cryptic to be meaningfully inspected by a human, and have more complex variable importance measures [2].

The relative interpretability of these non-linear models played a large factor in our decision to add random forests to our modeling stack at Sift Science, which you can read more about here: http://blog.siftscience.com/blog/2015/large-scale-decision-f...

[1]: http://statweb.stanford.edu/~tibs/ElemStatLearn/

[2]: http://www.massey.ac.nz/~mkjoy/pdf/Olden,Joy&DeathEM.pdf

Re: The Unreasonable Effectiveness of Random Forests

#33

Post by ajtulloch » Sun, Sep 20, 2015, 1:25 AM UTC

CyberDildonics wrote: post #25

ajtulloch wrote: post #3

The article mentions that "The main drawback of Random Forests is the model size. You could easily end up with a forest that takes hundreds of megabytes of memory and is slow to evaluate." A cool trick for speeding up trained random forests/gradient boosted decision trees is to dump the tree to C/ASM, compile it, and dlopen it as a function pointer. Depending on the model and the architecture, you can get an 8x speed…

Why go through all that though? The fundamental principle is that it is flattening the buffer to one linear section of memory, making the pointers jump to a location that is likely already prefetched and possibly in the same cache line. Dumping to C seems like it is missing the point. A library like Cereal should be able to do this almost trivially simply by serializing the data, unserializing it to a flat buffer, th…

That's actually not the case - that's one effect, sure, but when you compared the compiled trees to the flattened trees (the strategy just described), the compiled trees are substantially faster across a range of parameters. See http://tullo.ch/articles/decision-tree-evaluation/ for a detailed evaluation.

Re: The Unreasonable Effectiveness of Random Forests

#34

Post by CyberDildonics » Sun, Sep 20, 2015, 2:21 PM UTC

ajtulloch wrote: post #33

CyberDildonics wrote: post #25

Earlier quoted context omitted.

Why go through all that though? The fundamental principle is that it is flattening the buffer to one linear section of memory, making the pointers jump to a location that is likely already prefetched and possibly in the same cache line. Dumping to C seems like it is missing the point. A library like Cereal should be able to do this almost trivially simply by serializing the data, unserializing it to a flat buffer, th…

That's actually not the case - that's one effect, sure, but when you compared the compiled trees to the flattened trees (the strategy just described), the compiled trees are substantially faster across a range of parameters. See http://tullo.ch/articles/decision-tree-evaluation/ for a detailed evaluation.

That's really awesome that you put that together, but I have to think that 200 points and a depth Fundamentally I can't think of any reason compiling a tree would be a good approach in a general sense.

Another optimization is to put make splits multiple levels instead of just one. How many might depend on how many you can squeeze into a cache line. If you split position is a float and the dimension is a byte, each split would be 5 bytes. You can squeeze 12 of these into a cache line. You can go 3 level down instead of one by packing 7 of them into a chunk (1 split + 2 splits + 4 splits). This still leaves room for a 32 bit index or pointer for each of the 4 leaves.

Re: The Unreasonable Effectiveness of Random Forests

#35

Post by eternalban » Sun, Sep 20, 2015, 3:23 PM UTC

1971genocide wrote: post #17

eternalban wrote: post #11

aside: There really needs to be a law :) capping cpu cycles / information gained. Energy ain't free. Emissions are not harmless. OP appears to be a gas-guzzler algorithm.

What are you even talking about ??

The article. Did you read it all the way to the end?