Earlier quoted context omitted.
> We can't know for certain whether something is or isn't conscious, but we have to try to make a best guess. We now have AI which is trained on data generated by humans who are (probably?) conscious and feel pain etc. So it can generate output which implies that it's conscious and can feel pain etc., even when that isn't the case. What do you do with something that can pass the Turing test but still isn't human?
Thinking about the analogical inference helps here: We know that LLMs work very differently from humans, and we know why Sydney or LaMDA sometimes says it is conscious (it imitates human text). So the analogical inference doesn't allow us to infer that LLMs are conscious, or provides only very weak evidence. And since it seems plausible that most things (e.g. rocks) are not conscious, it seems reasonable to think tha…
The existing ones can write code. Suppose we get ones that can write better code and engage in self-improvement. Now you have something which is a billion lines of code and is under constant self-modification. We no longer have any idea how it works, but it can feign consciousness in much the same way as existing LLMs. What's the test for if it actually achieves consciousness?
I can give you one that a human being in good health will pass and a rock will fail. Can you give me one that an AI feigning consciousness will fail?