Related: Deep neural decision forests (ConvNets+Random Forests)
http://www.cv-foundation.org/openaccess/content_iccv_2015/pa...
http://matthewalunbrown.com/papers/1603.01250v1.pdf
https://topos-theory.github.io/deep-neural-decision-forests/
Deep Forest: Towards an Alternative to Deep Neural Networks
11–20 of 100 posts
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#12[deleted]
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#13XGBoost also appears to have a GPU implementation.
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#14"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train."
Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models that use BatchNorm nowadays, so generally tuning there is not an issue either. Many layers of 3x3 conv with stride 1 is still magical.
Basically: deep NNs can work pretty well with little to no tuning these days. The defaults just work.
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#15So if this works well why is there no comparison on ImageNet?
In the conclusion section the authors said that : "If we had stronger computational facilities, we would like to try big data and deeper forest, which is left for future work." and that: "As a seminar study, we have only explored a little in this direction."
If someone proposes a method as an alternative for a field, they need to test this method on the accepted benchmark dataset for that field. For object recognition in static images this dataset is the ImageNet competition. Computing power can be bought from AWS if no cluster is available. The lack of it can't be an argument.
Not saying that the paper has no reason to exist, I think it is generally well written and decision trees certainly deserve attention. If they can do representation learning on high level this is certainly something to look into. But it shouldn't claim to be an alternative to state-of-the-art deep learning if there is no data for this comparison. Everyone can solve MNIST (or even CIFAR).
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#16I don't know about the others, but the two visions dataset they compare to (MNIST and the face recognition one) are small datasets and the CNN they compare to doesn't seem very state of the art. It also seems each layer of random forest just concatenates a class distribution to the original feature vector. So this doesn't seem to get the same "hierarchy of features" benefit that you get in large-scale CNN and DNN.
To your point that they are comparing small datasets. I dont see that as a problem. If they achieve better results on small datasets that is a great achievement, as often the bottleneck is the size of the dataset rather than computation time.
> often the bottleneck is the size of the dataset rather than computation time
That's generally true for DNNs, which is a good place to be if you have lots of data. This typically isn't true for tree based approaches, which is why they fell out of fashion in some problem domains; they don't generalize as well. This paper doesn't seem to change what we already know in this respect.
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#17"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…
I couldn't disagree more. The defaults don't just work, and the architecture of the network could also be considered a hyper parameter in which case what would be a reasonable default for all the types of problems ANN are used for?
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#18"In contrast to deep neural networks which require great effort in hyper-parameter tuning, gcForest is much easier to train." Hyperparameter tuning is not as much of an issue with deep neural networks anymore. Thanks to BatchNorm and more robust optimization algorithms, most of the time you can simply use Adam with a default learning rate of 0.001 and do pretty well. Dropout is not even necessary with many models tha…
I agree with rkaplan. I've been working with many different visual problems and that comment is pretty consistent with what I've seen.
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#19XGBoost also appears to have a GPU implementation.
Yes, as a plugin, and on its way to make it to the trunk AFAIK. I've integrated it into deepdetect recently because even as a beta it works well and complements GPU based DNNs fairly naturally. Deep learning practitioners are already well equipped with GPUs so having XGBoost run on them as well is a good bonus!
Re: Deep Forest: Towards an Alternative to Deep Neural Networks
#20Would like to see three things coming out of this
1. R code implementation (could probably write this myself but would make things easier)
2. How to get feature importance? Otherwise difficult to implement in business context.
3. Better benchmarks