> Probably the most concrete of this is "Fall; or Dodge in Hell" which involves a simulation of people's scanned brains rediscovering qualia and constructing their own simulated world from scratch.
This is a great example for a discussion about The Hard Problem. Here is the description from the book of the inner experience of this scanned brain as it gets booted up:
> What came next could not, of course, be described without using words. But that was deceptive in a way since he no longer had words. Nor did he have memories, or coherent thoughts, or any other way to describe or think about the qualia he was experiencing. And those qualia were of miserably low quality. To the extent he was seeing, he was seeing incoherent patterns of fluctuating light. For people of a certain age, the closest descriptor for this was “static”: the sheets, waves, and bands of noise that had covered the screens of malfunctioning television sets. Static, in a sense, wasn’t real. It was simply what you got out of a system when it was unable to lock on to any strong signal—“Strong” meaning actually conveying useful, or at least understandable, information. Modern computer screens were smart enough to just shut down, or put up an error message, when the signal was lost. Old analog sets had no choice but to display something. The electron beam was forever scanning, a mindless beacon, and if you fed it nothing else it would produce a visual map of whatever was contingently banging around in its circuitry: some garbled mix of electrical noise from Mom’s vacuum cleaner, Dad’s shaver, solar flares, stray transmissions caroming off the ionosphere, and whatever happened when little feedback loops on the circuit board got out of hand. Likewise, to the extent he was hearing anything, it was just an inchoate hiss.
The Hard Problem asks: who is experiencing this qualia and why is there an experiencer at all? Stephenson writes how this simulated brain is experiencing static as it condenses into meaningful patterns, but he implicitly starts with someone experiencing this static qualia. If this is the very beginning of the simulated brain booting up, where did this experiencer come from?
> At some point, we'll have AGI that pass any test we can think of and we'll still have people arguing that these cannot be conscious.
Because you're conflating intelligence with consciousness. There is no test for consciousness. In fact, you can't even prove that other human beings are conscious, you only know that you yourself are because it's self evident to you (cogito ergo sum, I think therefore I am). The whole point of the hard problem is that you can imagine something exactly like a human being that passes every test of being a human being (e.g. an AI) but still not be sure that it has any inner experience.