Could someone explain in layman words what is the qualitative contribution of the paper and why it is important?
Sure. Random forests are a very powerful machine learning method that has proven to give excellent performance with very little tuning. Neural networks with many layers (aka deep learning) is the new frontier of machine learning - they are very powerful but require extensive tuning (the architecture of the network) and lots of examples before they become practical. In this paper, the authors combine the two methods -…
It's not so much that they stack a random forest on top of a neural network, as that they are trying to train neural networks that have tree-like properties. Deep neural networks can be expensive to evaluate because every layer is needed to compute the resulting label. On the other hand, in a decision tree you only need to evaluate one of a nodes children (i.e., evaluation is linear in the depth of the tree).
They combine these ideas by allowing some nodes in the network to branch (so for a given image, you choose one of two neural networks to evaluate). These nodes don't have to occur at the top level, but can occur in the middle of the network as well.
The difficulty is how you train these nodes; that's where the differential technique for optimizing trees come in. Because you have a unified training algorithm, you can mix and match tree-like nodes and neural network like nodes and train the whole thing.
All in all, the hope is that getting a label for a given node is much faster, so you can run these on smaller devices such as phones without sacrificing accuracy.