Earlier quoted context omitted.
I take the point to be that there aren't "deeper" fundamental principles at play in these models. Tremendous progress has comes from simply tweaking of the numbers of layers, or how the feed forward to each other (skipping layers, etc), or by throwing more computer power or data at the same basic algorithm. Where might we look for deeper principles? One idea is to consider what brains do and how they might be doing i…
As a physicist judging from the outside, I share some your feeling. Are there general laws governing "learning"? Theorems? Are there "deeper" things to learn as humans? The thing is people in the field don't need heavy intuition or math. In some ways that's good (if you just want a result to utilize) and in others, it's bad (if you are a curious person).
This has been my question too...
Based on my understanding, it all really boils down to probability, statistics and a few important theories like Vapnik-Chervonenkis which provides the mathematical foundation for what "learning" is, can we even learn from the given data and how well can we learn (VC dimension, etc).
But I would love it if someone can point me to or explain / derive from core first principles the concept of "learning".