Perhaps I am unimaginative about whatever AGI might be, but it so often feels to me like predictions are more based on sci-fi than observation. The theorized AI is some anthropomorphization of a 1960s mainframe: you tell it what to do and it executes that exactly with precise logic and no understanding of nuance or ambiguity. Maybe it is evil. The SOTA in AI at the moment is very good at nuance and ambiguity but some…
I've listened to some old sci-fi radio shows and it's interesting how often "the computer never makes a mistake" comes up. Which is usually followed by the computer making a mistake.
In any non-edge case (that is, where the system is operating in ideal conditions and no flaw or bug, known or unknown, exists in the system), a verifiably functioning computer will produce the exact same results for any process every time.
If the computer does not do what you expected it to do and spits out garbage, then you gave it garbage data.