Earlier quoted context omitted.
If we can't understand the problem, will we be able to understand the solution presented by that AI? Or we'd just apply it, trusting blindly the unfathomable reasons the AI used? Do we have an AI where the decision tree can be grasped by humans?
It seems inevitable to me the moment AI capacity seriously surpasses human (as a whole) capacity in any specific topic, it becomes an oracle. I hear of efforts to translate machine decision making to human understandable terms, but if it is a question of raw intelligence, it will quickly become impossible to understand.
And to be honest, I wondered about that with GPT-3. Maybe it could give a more profound answer to a given prompt, but it chose not to, since it found the prompt to be too silly, and it responded in kind. Just like adults are able to entertain an imaginary universe of children.
So even to explain that we want a "serious" answer might be difficult.