Consciousness is not computation

So let me expand a bit of why I used the adder circuit as an example.

There's nothing about addition that can say "this is addition" in an objective sense. We are dealing with binary encoding of numbers. There's nearly an infinite number of ways to encode information. The design of any circuit deals with particular encoding of information. This extends all the way up to the higher levels. There are an infinite number of ways to encode the state of a go board. Any AI that plays go will deal with some specific ways of encoding the state of the board, and that will be both its input and output.

The most popular "intuitive" model of how consciousness arises is basically hand waving about a system inspecting itself.

But you have to understand, a system inspecting its own state is not any different from a system inspecting any other arbitrary state. For the state of the system will be encoded in some way that was chosen arbitrarily. There's nothing essential about any encoding of any state that can possibly give raise to a phenomenon like consciousness.

You can come study the behavior of any arbitrary system, given certain inputs and outputs, and assign any meaning you want to its states. You can retro-fit any intepretation of state to the system.

Because the thing is with encoding information is it can be completely arbitrary. I can interpret the state [01010101] to be my username, and then interpret [1111000111] to be my real life name. This would allow me to say that a system that takes 01010101 as input and produces 1111000111 is actually a system that can derive my real life name from my username.

I don't see how this addresses the key point -- those aren't random distinct states; there's an explicit description of how one state transitions to the next. That's part of the definition of the Turing machine!

If you could describe a mapping from the state of your iron bar to a Turing machine's state, such that the bar correctly transitions from one machine state to the next, then yes, it is a computer! Although as others have noticed, you'd need a ludicrously large and complex mapping; the infinitesimal tail would be wagging the infinite dog. At that point you could certainly argue that the computation is really happening in the mapping itself.

You cite the primacy of sensation, but what more is sensation than a message from the sensory organ to the brain. Perhaps encoded in the brains internal language rather than the plain text we are used to, but messages none the less. The concept of existence-as-messages is thus not contradicted by the experience of messages from your own body.

Irrespective of any underlying physical reality, I can't escape the fact that I can never test reality itself, only my perception of it. We can get a lot of mileage out of the physicalist approach, accepting as an axiom that what we perceive as reality is real. But in that ground up approach we've had terrible difficulty deriving our ourselves from physical first principles. I'm not saying its impossible or wrong to take the path from atoms on upwards to consciousness and perception. But no one has been able to make it to the end of that path. What happens if we go the other way instead, starting from the known conclusion "I'm here and conscious enough to converse" and working our way down to the perception/understanding of an inanimate reality governed by objective physical principles? Can a system only designed to represent/reason about people talking to and about people be hacked into a system for representing and reasoning in general?

If we start from purely language models like GPT3 and continue to teach it the "social reasoning" of saying things we want to hear, will the language model eventually become capable of non-social reasoning as well? In the process of figuring out what we want to hear well enough to describe a non-contradictory scene to us, does GPT3 have to actually learn the rules of 3d euclidean space governing the scene? Is there any possible way to avoid scene contradictions without a full understanding of the underlying physical reality it is supposed to describe to us? If a camera feeds data to a language model, does the language model suddenly have eyes? What about pictures from the internet?