Earlier quoted context omitted.
> I assume you'll agree that we'll still be able to observe what it does do and understand that it's doing it No. To make that leap, I must assume the conclusion. I cannot know that I'll be able to observe what it does, because I don't know what it will do (or whether I can even perceive the dimension in which it does it for that matter). Secondly, the word "understand" is subjective. I assume that Musk meant the wor…
I had a really long reply written, but before posting I went to try and find the source of the Elon Musk paraphrased belief, and I can't find it. I don't know where sixQuarks got it from, but without knowing what the source is, it's hard to second-guess what Elon Musk was actually trying to say (or whether he even said it at all). That said, ignoring the Elon Musk bit, even if a superintelligent A.I. figures out how…
And here is the actual paragraph:
"One topic I disagreed with him on is the nature of consciousness. I think of consciousness as a smooth spectrum. To me, what we experience as consciousness is just what it feels like to be human-level intelligent. We’re smarter, and “more conscious” than an ape, who is more conscious than a chicken, etc. And an alien much smarter than us would be to us as we are to an ape (or an ant) in every way. We talked about this, and Musk seemed convinced that human-level consciousness is a black-and-white thing—that it’s like a switch that flips on at some point in the evolutionary process and that no other animals share. He doesn’t buy the “ants : humans :: humans : [a much smarter extra-terrestrial]” thing, believing that humans are weak computers and that something smarter than humans would just be a stronger computer, not something so beyond us we couldn’t even fathom its existence."