Superintelligence cannot be contained: Lessons from Computability Theory

Whether the system "works" or "doesn't work" is dependent on what the machine was designed to do, which is not an objective physical fact about the machine. Perhaps the machine was not meant to open the door when an object is detected, but to close it instead, or to do something else entirely; only the designer would be able to tell you one way or the other.

The same is true for all computation, and that is Searle's point.

> A computer (the theoretical model) is, be definition, something that can perform coherent reasoning without any special internal state.

Computers don't actually engage in reasoning, though, for the same reason. A machine is just a physical process, and physical processes do not have determinate semantic content.

Ross and Feser then argue that because thoughts do have determinate semantic content, they are necessarily immaterial, and I think they are correct.

(This argument is unrelated to qualia; I don't think qualia are fundamental to reason itself.)

This property of determinacy seems ill defined as well. It's basically defined from the assumption that the human mind is immaterial. If a machine and a human both arrive at the same result when posed a question (say, they both produce some sound that you interpret as meaning '42'), by what measure can you claim that one had semantic meaning and the other did not?

The idea of cognitivism is that there is no fundamental difference (even though of course it is very likely that the process by which this particular machine arrived at that result is different from the process by which the human did).

If I stand by a door and open it when big objects come into my field of view, how is that different from a machine doing the same?

And then, if I had a Machine that could converse and act just like a human (including describing its feelings and internal sensations) while doing nothing fundamentally different from our current PCs, by what measure would you say that this machine is 'simualting' a mind and is not in fact a mind in itself? (though of course it would be a different mind than a human would have).

Can a sensation exist without neurons firing? The root of our conversation is the question if a sensation purely exists in the physical world. If it does, then it is possible to measure it. If it doesn't, then that breaks our scientific understanding of the world and would be exciting news.

> They're not the same, for what it's worth. The term "flashlight" conveys a certain intent and structure that "photons traveling away from a light bulb with a battery and a current" does not.

Yes there is no strict definition of a flashlight. Let's use your definition of a flashlight. Is it possible in your mind to separate the concept of a flashlight and your definition? Without "your definition here", the flashlight no longer exists. My point was without firing neurons, the sensation does not exist.

> The fact that two different phenomena are closely coupled via a cause and effect relationship does not make them the same phenomena.

My wording was not the best. My point was that the sensation of red is physically equivalent to neurons firing. How do we measure a sensation? If we cannot measure a sensation, does it exist in the physical world? If it doesn't exist in the physical world, then what does it existence mean to the scientific community?

> If you push two magnets together, the fact that the same force causes them to attract or repel does not mean that the motion of the first is literally equivalent to the motion of the second, or that the force itself is literally equivalent to either motion. They are closely correlated, but ultimately distinct.

I agree that these forces are distinct. We can measure the force of each magnet separately and we can define the motion of one magnet without referencing the motion of the other.

> You just can't avoid the fact that qualitative phenomena do exist in their own right. They can't be explained away using a physical model that assumes from the get go that they don't exist.

What is a qualitative phenomena? I couldn't find information on this term.

If we can't measure a qualitative phenomena in the physical space, what does it mean to exist?

This paper defines the term "containment problem" in a nonstandard way. This is different from how I use it in https://arxiv.org/pdf/1604.00545.pdf (which is largely about preventing side-channels and security vulnerabilities in realistic simulator environments) or how Bostrom's uses it.

In this paper, the containment problem is defined as executing a Turing machine for unbounded steps, iff it does not cause harm, given a function which defines whether an output counts as harm. They use a Godelian argument to show that this is incomputable, because a program could simulate the simulator, and condition its behavior in a way that creates a contradiction.

This is a toy problem which sort of vaguely rhymes with AI containment and AI safety, but contributes nothing to either of them. A real-world simulation environment wouldn't have infinite computation; there would be a finite bound, and "the AI reached the end of the simulation period without making up its mind about what it would do" is just an expected result which means the AI hasn't been proven safe.