Great post. And that's exactly where I think we are with language models... we as a civilization are hypnotized and enchanted by the overfitting of models whose parameters are beyond our understanding, but whose mistakes we are more likely to forget than its accuracies, which again is a central human characteristic that explains our attraction to both psychics and slot machines.
Heck, it even explains my own attraction to overfit sports betting algorithms. No one is immune.
What's dangerous is when something like that replaces independent thought and becomes societally pervasive. That's an "oracle" the likes of which ancient civilizations warned that believing would lead to tragedy (or at a minimum, accidentally boning your own mother).
I'm an atheist, but raised Jewish. I read the Torah as a series of specific warnings and prohibitions against every type of shamanism, magic, witchcraft, prognostication, and deification of systems which predict (as well as systems which attempt to turn language into machinery, and worship the machine they've built ... see also, "Sound of Silence" by Paul Simon and "The Future" by Leonard Cohen, which both express this theme well). The framework requiring proof and disavowing illusion or the belief that all is illusory is notably different from a Buddhist perspective, for example.
We as a culture, right now, are not handling well the rise of a golden idol or an oracle in our midst. The right response is to try to trace the output back to ground truth and figure out why your model made a prediction... or else to build a model from ground truth and see how it performs against the oracle. We are doing neither. We're diving headlong into our own confirmation biases.
[Edit] I just wanted to add, because I got off track, that your conclusion about what's going on with human curiosity in cases where prediction is not the issue seems right to me. Barring some edge cases like predicting an eclipse and using it to slaughter your enemies, I think a lot of us do simply want to understand how things work, because figuring them out is enormously gratifying and is the work of lifetimes of incredible people who came before us. Using that knowledge or those techniques to predict things is technology, not science, and while I'm a fan of both, the former is only ever a practical test of the latter. Moreover, the sense of accomplishment of randomly walking your way to a profitable model is ephemeral and in a way earthbound, limited to the plane of one's own brief existence. Even if it were platonically perfect, a model is only saying how something behaves, not how it works. That's nothing compared to the joy of figuring out even the most trivial or axiomatic thing about how a cell or a compound or a physical structure or anything works, about how the universe actually works. And I think our better angels tell us to seek those answers, because our own life is fleeting, and predicting behavior is, like wealth, something you can't take with you. And not something you'll be remembered for anyway.