Random Forests for Complete Beginners

Random Forests for Complete Beginners
victorzhou.com

They kinda cover it in the section in 3.1 and 3.2 with Bagging and Bagging -> RandomForest, but it'd be good for them to explain Boosted Trees here as well.

As far as I understand it, random forests are an aggregation of trained trees based on randomly sampled data points from the original data set. It doesn't necessarily make them more accurate on the training dataset, but it makes them more generalised and less likely to overfit (https://en.wikipedia.org/wiki/Overfitting), because the different trees are likely to focus on different characteristics of the dataset.

Boosted trees do become more accurate, as they resample, but give more priority to data points that weren't correctly classified by the earlier models.

This is pretty concise but mostly for decision tree but only half of it.

CART is the framework for decision tree for classification and regression. This article only addresses the Classification part which usually use Gini which is a class of split that split along parallel axises (there are oblique trees). The regression part uses more traditional statistical linear regression to calculate split point (SSTO).

Very light on Random Forests though, doesn't talk about out of bagging, implication of bootstrap and ordinal data, etc.. but overall I think it's a neat introduction.

> we only try a subset of the features,

You bootstrap features without replacement at every split. Instead of just bootstrapping rows/observations like in bagging.

The concept of weak learners ensemble together to become a strong learner is done by Dr. Ho work under her paper Random Subspace where she does it with decision tree and basically proposed Random Forest before Dr. Leo Breiman (both independently came to Random Forest). Her advisor have the theoretical paper for proof of weak learner, stocastic discrimination.

That's the point where you should backtrack.

Since this is a blog post it may not be kosher copacetic on all the details. But if you're studying from books and papers all the notation will either be way too well-known (set theory, cartesian products, R^d vector spaces, lp/Lp norms) or explicitly explained.

If you're behind or fuzzy on the more basic stuff (what's an equivalence class? What's a Cartesian product?) I recommend the first two chapters in Munkres' book of topology. The book builds pretty far out into uncharted territory, but its recap of the basics is rigorous and superbly explained with copious illuminating prose.