Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Random Forests for Complete Beginners

victorzhou.com

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Random Forests for Complete Beginners

#31

Post by GaryNumanVevo »

vzhou842 wrote: post #15
anthony_doan wrote: post #7

My thesis is related to trees, forests, and ensemble of forests. This is pretty concise but mostly for decision tree but only half of it. CART is the framework for decision tree for classification and regression. This article only addresses the Classification part which usually use Gini which is a class of split that split along parallel axises (there are oblique trees). The regression part uses more traditional stat…

Hey! Author here. Appreciate the feedback. You sound like you know what you're talking about. My site is open source, and if you want you can make a few edits to this article and submit a pull request! https://github.com/vzhou842/victorzhou.com/blob/master/conte...

honest question: who does this benefit but yourself? I don't see a commit log or blame on your site for who wrote what.

Aren't you effectively claiming authorship for other's work?

Re: Random Forests for Complete Beginners

#32

Post by platz »

claytonjy wrote: post #27
b_tterc_p wrote: post #12

While there’s nothing wrong with random forests, they’re a bit of a red flag for me as they’re easy to implement without any real understanding of what’s going on. A lot of junior data scientists just default to saying random forest to solve any problem because it tends to have the most predictive power of the models they’re comfortable with. That’s a bad sign.

What's the bad sign there? If juniors are reaching for RF as a first-pass over say, (logistic) regression, that seems like a _great_ sign; no other approach has such a high average performance-effort ratio. For non-huge, naturally tabular problems, it usually takes me 10x or more the effort to beat the very first RF I train. Developer time matters! If they start going to GBM's or neural nets first...I'd call _that_ a…

I can understand why a neural networks first is bad, but I don't understand the problem with GBM's

Re: Random Forests for Complete Beginners

#33

Post by khawkins »

djaychela wrote: post #2

I've started studying ML (as someone who's hoping to transition at 47 to a different career from teaching and music), and some concepts I've found easy to get, others not so much (I'm old, so give me a break!). Often I need to have a concept explained in a different manner before I'll have a 'Oh, I see!' moment, and because I don't have a great grasp on the maths needed (which I'm working on, but it's s-l-o-w), as so…

If you understand how CART decision trees are trained, then you can see why random forests are powerful by thinking through their training process.

In the first step of the decision tree training we pick the best feature split, divide the data into two groups, then train each of the two data groups independently. But what about the 2nd-best feature split? In some sense, we lose the information the other splits could provide.

To see this, when testing queries, the first step is to look at that best split and pass the query to one of the two sub-trees. But those trees have only been trained with half of the training set data, and thus have weaker discriminatory power. Every split down the tree has diminishing returns in terms of how much information it provides.

Now think about what the random forest does. If the feature which contains the best split is chosen for a particular tree, the split will be the same. But if it doesn't, then if the feature for the second split is present then it will be chosen. If the top 2 features aren't present then the third best split will be chosen, and so on.

Thus, across our forest we have representatives of a range of feature-splits, each trained on more data and thus have more discriminatory power per split. The aggregation step at the end combines the information gleaned from these different models. Each one of them is weaker than the original CART decision tree, but has gotten more information out of the data for the features it was given. Thus, together, they are much better predictors than by themselves.

Re: Random Forests for Complete Beginners

#34

Post by anthony_doan »

savagedata wrote: post #22
sandGorgon wrote: post #10

Earlier quoted context omitted.

is there a better tutorial/course for a beginner into this field ? the end goal not being academia, but being able to think and write reasonable production code.

I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedb…

It's a really good tutorial.

I like how you talk about Conditional Inference. My thesis is suppose to overcome the brute force of exhaustive search for best splits that Random Forest does (I use Dr. Loh's GUIDE trees) using statistical methods.

> Many implementations of random forest default to 1/3 of your predictor variables.

This is interesting. I hear it was sqroot(total number of predictors).

> Ensemble methods combine many individual trees to create one better, more stable model.

I think stable can be more clarify to having good training accuracy and low generalize error (unseen data error rate) compare to individual tree. This is what Dr. Ho talk about with forest.

But other than that I think it's an awesome tutorial.

I've seen what other tree and forest do for better generalization with unseen data is pruning is using CV and choosing 0.5 to 1.0 std error as a cut off point. That may be a thing to talk about if you are interested in that.

Re: Random Forests for Complete Beginners

#35

Post by vzhou842 »

GaryNumanVevo wrote: post #31
vzhou842 wrote: post #15

Earlier quoted context omitted.

Hey! Author here. Appreciate the feedback. You sound like you know what you're talking about. My site is open source, and if you want you can make a few edits to this article and submit a pull request! https://github.com/vzhou842/victorzhou.com/blob/master/conte...

honest question: who does this benefit but yourself? I don't see a commit log or blame on your site for who wrote what. Aren't you effectively claiming authorship for other's work?

My blog is still pretty new, so I haven't had to think about this situation yet - I'm the only one who's committed so far. If someone were to contribute I'd be more than happy to figure out a fair way to attribute them, though!

Re: Random Forests for Complete Beginners

#36

Post by vzhou842 »

rrggrr wrote: post #24
vzhou842 wrote: post #16

Hey, Author here. If you're new to ML you might also like my introduction to Neural Networks: https://victorzhou.com/blog/intro-to-neural-networks/ Discussion of my neural networks post on HN: https://news.ycombinator.com/item?id=19320217

Great contributions. Great work and I ought not ask for more, but gosh if you could put in real world examples with code it would be great.

Thanks!

When you say "real world examples", what do you have in mind? A lot of "real world" uses of random forests are basically just directly calling scikit-learn or something like that. In my neural network post, I implemented a simple neural network from scratch because I felt like it'd be valuable for beginners, but I wouldn't call that "real world".

Re: Random Forests for Complete Beginners

#37

Post by sandGorgon »

savagedata wrote: post #22
sandGorgon wrote: post #10

Earlier quoted context omitted.

is there a better tutorial/course for a beginner into this field ? the end goal not being academia, but being able to think and write reasonable production code.

I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedb…

thanks that's pretty cool !

Re: Random Forests for Complete Beginners

#38

Post by savagedata »

anthony_doan wrote: post #34
savagedata wrote: post #22

Earlier quoted context omitted.

I wrote this regression tree tutorial a few years back that might be a good complement to the tutorial above since it covers regression instead of classification and goes on to talk about bagging vs random forest, out-of-bag samples, and tuning parameters: https://github.com/savagedata/regression-tree-tutorial I wrote it at the start of my career and haven't shared it beyond my study group, so I'm happy to hear feedb…

It's a really good tutorial. I like how you talk about Conditional Inference. My thesis is suppose to overcome the brute force of exhaustive search for best splits that Random Forest does (I use Dr. Loh's GUIDE trees) using statistical methods. > Many implementations of random forest default to 1/3 of your predictor variables. This is interesting. I hear it was sqroot(total number of predictors). > Ensemble methods c…

Thank you for the useful feedback! I'll have to look up GUIDE trees.

> This is interesting. I hear it was sqroot(total number of predictors).

I was probably looking at the randomForest R package documentation [1], which says:

> mtry Number of variables randomly sampled as candidates at each split. Note that the default values are different for classification (sqrt(p) where p is number of variables in x) and regression (p/3)

I checked the H2O implementation of random forest [2] and they use the same defaults.

I'll add a note about the one third default being specific to regression since that seems like an important distinction.

[1] https://www.rdocumentation.org/packages/randomForest/version...

[2] http://docs.h2o.ai/h2o/latest-stable/h2o-docs/data-science/d...

Re: Random Forests for Complete Beginners

#39

Post by chrisshroba »

GaryNumanVevo wrote: post #31
vzhou842 wrote: post #15

Earlier quoted context omitted.

Hey! Author here. Appreciate the feedback. You sound like you know what you're talking about. My site is open source, and if you want you can make a few edits to this article and submit a pull request! https://github.com/vzhou842/victorzhou.com/blob/master/conte...

honest question: who does this benefit but yourself? I don't see a commit log or blame on your site for who wrote what. Aren't you effectively claiming authorship for other's work?

I think this assumes writing an article only benefits the author, when in fact I think it benefits readers far more (if the blog isn't monetized, the author only really gets satisfaction out of it). If I had expertise in some area I would be happy to contribute to someone else's blog, provided I were properly credited (which the author stated he would do).
Post reply on HN