> Personally, I'm more interested in analyzing those black boxes than tinkering ones that "seems to work", would it be with graph theory, analysis, etc.
It's not that nobody is doing this, it's that we are making very slow advances in our understanding. There are many, many, many different angles you could take and there's so far not a clear answer which one is right. We are uncovering many interesting properties, but some just open more doors than they close. For example, the whole adversarial research (crafting inputs that are misclassified despite being similar to a correct classified example) did not lead to a quick fix but to a whole research area where we are still even trying to understand what exactly we are dealing with. It's not so much that everything is black magic and nobody cares about why the problem is really so hard, but it's instead just hard to to make progress.
> To me, if something works but we're unable to really understand why it does, it's more the realm of "testing broken clocks that work twice a day".
That's not fair. We have extensive empirical tests, it's really been enough time and enough eyes to see that it's not some kind of coincidence or luck but instead we really have uncovered a way to make progress we haven't dreamed about before. It's not a "broken clock works twice a day" situation, we are pretty sure of that.
I understand the frustration, but I can assure you that research is totally aware of NNs pitfalls and our limitations. I can imagine that this situation is weird for someone not into ML. That to the ordinary person, the world seems pretty solved. We understand the basics of physics, chemistry etc. Now here's something where we really lack the basics, even some "newtonian physics", we use something without having a guiding theory. That's so 19th century! As someone interested in theory and empirical basics to understand the properties, I can tell you it's just really hard. Our usual mathematical tools and theories do not exactly fit, from optimization to statistical learning theory. What we do just doesn't make that much sense, but the improvements suggested by these existing theories lead to worse performance. We try to come up with new ideas and new perspective on the different problems, but so far we've not found a promising candidate. It's like early science but it's still scientific.