>So what context would cause me to seriously consider the possibility that engineers had created a computer program that is conscious and an intentional user of language? Let me outline one potential sequence of steps. The first requirement is that the computer program has a body (either physical or virtual) and sense organs; there are many reasons for this, but for the purposes of this discussion the most relevant one is the fact that without a body, a computer program could have no desires or emotions, and I believe desires and emotions are necessary for consciousness. Then I’d want to see an embodied agent that could navigate its environment in order to survive as well as, say, a lizard can (and as a point of comparison, certain iguanas can live for decades in the wild). Next I would want to see an embodied agent with the same capacity to deal with novel situations as a mouse. After that I’d want to see agents whose social dynamics are as complex as those of wolves, and then agents with the tool-making abilities of chimpanzees. At that point I would want to see people successfully teaching such embodied agents how to communicate their desires, perhaps by using a button board or some other nonlinguistic modality, the way that people have taught chimpanzees and domesticated dogs.
I agree with some parts of this piece, but paragraphs like this one above seem pretty uninspired and simplistic. It's entirely plausible that a conscious mind would not be evolutionarily incentivized to be able to do those things. ie just because animals on earth needed to develop specific talents doesn't mean that other conscious entities need to. Why would a computer program need to hunt for food like a mouse would? Making tools like chimp? these seem like nonsensical metrics.