Well, it is definitely something but it being the "Breakthrough of the Decade" seems pretty unlikely to me (given my available evidence). I do not know well other examples beyond case of Automatic Speech Recognition, but since this case caused a lot of noise, I bet it is responsible for a reasonable chunk of the Deep learning "buzz". Here is my take about this. If you look at papers from Microsoft like Seide et al 20…
The thing that NNs have in their favor that other "20 year old techniques" lack is their ability to model any mathematical equation. There is no fundamental limit to the complexity of systems NNs can model (as there is with other AI techniques). The problem with NNs is the difficulty of training them. Back propagation with random initial weights is simple, but it can easily converge on suboptimal local maximum if the…
I'm sure that a decision tree can also be viewed as a [universal approximator](http://en.wikipedia.org/wiki/Universal_approximation_theorem) if you let tree height go to infinity (just as you need to let layer size grow unbounded with a NN). In practice, this power is at best irrelevant and often actually a liability (you have to control model complexity to prevent overfitting/memorization).
And, importantly, being able to theoretically encode any function within your model is not the same as having a robust learning algorithm that will actually infer those particular weights from a sample of input/output data.