Deep Neural Networks learn layers of linear functions - represented by adjustable 'learning' weighted connections.
Each layer of neuron activations is a new representation of the data - produced by the weighted connections.
Function compositions of linear functions are still only a linear function.
Each neuron in a layer sums it's weighted inputs this summation is a non-linearity that allows layers to be composed - function composition.
This is famously expressed in Minsky and Papert's 1968 'Perceptron': a single layer of network weights is incapable of learning XOR.
One analysis is that Neural nets transform the shape of the data until a single line on a twisty high dimensional manifold produces the desired distinction.
http://colah.github.io/posts/2014-03-NN-Manifolds-Topology/
A single layer network is a universal approximator and a net can be trained or distilled from another net - but deep nets are overwhelmingly better at the initial learning and discovery.
Neural nets have been related to Kadanof's spin glasses, suggesting learning is alike to phase transitions or sand pile collapse where small local changes can produce profound global changes.
https://charlesmartin14.wordpress.com/2015/04/01/why-deep-le...
Generally when training nets the learning initially learns a lot very quickly, most of the error vanishes at the start.
Word2Vec demonstrates that nets learn very powerful representations, that word2vec vector algebra is semantic points to unexpectedly powerful representations.
Similar semantic vector math can be performed on images and the same vectors can translate modalities, e.g. text to images.
https://github.com/Newmu/dcgan_code#arithmetic-on-faces
Natural Evolution produces efficient solutions.
I propose the successes of deep learning so far are partially explicable because they are working within human culture and perception, relearning our very efficient solutions - akin to distilling a deepnet into a shallow one.
This hypothesis will be tested if embodied deep neural nets using re-inforcement learning discover their own efficient solutions to performing tasks in the real world - robotics.
IMHO Peter Abeel's deep net, learning to robustly output robot motor torques directly from camera pixels will show if embodied deep nets can do discovery rather than relearning what we know. http://www.cs.berkeley.edu/~pabbeel/research_rll.html