My 0.02... I'm not a neural network expert but have done a fair amount of nonlinear dynamic modeling and large-system optimization work in my career. I have always felt (and commented on HH) that neural network based approaches to complex problems would work well up to a point and then fundamentally not be able to go any further because of the massive degrees of freedom in NN-based modeling and the emergent erratic, non-linear behavior. So a specific NN model could get me 95% accuracy or "performance" on some task, but 98%, or 99.5%, is mathematically impossible... limit of "impossibility" being different for each problem.
Trained neural networks of reasonable size have many thousands of independent variables and, as a result, have incredibly complex dynamics and behavior. Conversely, this explains why neural networks (of sometimes modest size) are able to approximately model behavior of extremely complex systems, linear or non-linear.
We can often understand the sensitivity of the NN to a specific variable in the NN, or the interplay of three variables together. Once we need to consider tens or hundreds of variables, we really can't fathom the dynamics and interplay of variables in the model any more... in some cases, iterative numerical sensitivity analysis can help. But even reasonably small NN's can have millions of variables and these numerical analysis methods become intractable to apply.
Remember the news of about 'single pixel' attacks against machine learning models that analyze and "categorize" images... researcher could change the category of an image by changing one pixel, the "right pixel", or hot pink or something. The researchers where in effect exploring a small sub-space of the NN and finding "regions of extreme dynamics" (highly weird behavior) in the model. There is no "rationale" for changing a cat to a dog based on one pixel going hot-pink in a 2M pixel... but there are extreme dynamics in model that allow that to happen.
I worked for a time on infrared detector development and "badly behaving" pixels (blinkers, etc.) were enemy #1 for IR computer vision because the wrong blinking pixel in a 4M+ pixel imaging array could cause all kinds of problems in the CV algorithms. It was okay for a pixel to be dead, or hot, or have low quantum efficiency... but pixels that blink, flicker, etc. periodically caused extremely erratic, unpredictable behavior of video analysis algorithms, which were largely NN-based, or used other high-degree of freedom modeling / decision-making methods.
While we can't predict this kind of weird behavior ahead of time from a specific NN model of reasonable size, we expect it. As briefly discussed above, we can't inspect complex NN models and figure out how and where they are going wrong. We'd have to perform an exhaustive, permutation-based search of the NN dynamic 'phase space' to determine what variable tweaks in the NN achieve the affects we are looking for... that search is, I think, a NP-hard problem (but I'm not sure).
There are a lot of other factors influencing the general "hardness" of modeling complex things with neural networks (data quality, etc.)... but this idea of "complex dynamics" in NN's is, I think, the fundamental problem that can be partially mitigated but not completely solved.