Earlier quoted context omitted.
I took a "Deep Learning" CS class in college back when it was in its early stages. I doubt the field is still called that now, but it was the subset of ML that has been rebranded as AI; includes LLMs, image generation, image recognition, etc. Like any class, it was confusing at first, but when I eventually grasped the math behind what we were doing, and of course the visual representations of different elements to sh…
"knowing what's going on" is a very loosely defined term here, it has very little predictive power where tipping points are going to show emergent behavior. Kind of like saying we can observe a single neuron and all of its chemical and electrical reactions, but that tells us very little about the emergent system state. And the entire state of the algorithm at any given point is what is important. To use a recent exam…
If I could summarize your argument, it sounds like you are saying that we can't understand the algorithm because we can't predict the output as it grows in size. While I agree that the larger the algorithm, the less predictable the output, I don't think this negates understanding of the algorithm itself.
I can build a slot machine, know how it works, but still not be able to reliably know who to cut in line so I can guarantee that I'll get the next winning pull. Just because I can't predict the output, doesn't make something not understandable. Same with all statistical behaviors. Quantum physics limits what we can measure but we still have math for it and understand why we can only measure speed or position, but not both, no mystery here if you look at the math behind it.
My slot machine doesn't hack other companies though, but only because it isn't connected, just like AI currently can't hack my mechanical dishwasher. If my slots were connected to the internet, could make api calls, and further had agentic capacity in some way, adding these features but being no different in stochastic attributation, it would be no different.