Artificial intelligence is not conscious – Ted Chiang

> If we give an LLM a prompt that reads “The following is a conversation between Julius Caesar and Genghis Khan,” it will generate a coherent dialogue between the two historical figures. But no matter how detailed the responses are, no matter how vividly they recount their respective historical accomplishments, we would never conclude that the LLM has conjured up digital re-creations of Julius Caesar and Genghis Khan, nor would we suggest that the historical figures are conscious

They might be in principle. It could be that the best way to generate a plausible dialogue is to bring up re-creations of the characters and have them act it out. LLMs definitely have been demonstrated to have world models in some cases. That helps generating text.

> presently I can not even fathom how they can be compared to the complexity of a system such as the human brain

Totally understandable; I don't think we can fully understand the human brain, using the human brain. We can understand its principles (firings and chemistry, structure and specialized areas, etc) but otherwise it's a capacity problem.

And while I can't fully understand myself, let alone another person, I definitely enjoy talking with people and sharing thoughts that I realize I wouldn't have had on my own.

The more I talk to people about consciousness and personal experience, the more I notice that most people hold some very strong unquestioned assumptions about these concepts.

The first and most frustrating confusion that comes up repeatedly in these discussions is the conflation between terms. For instance, people will use the terms "sentience" and "consciousness" interchangeably with "capable of having an experience" and "capable of high-level awareness of self".

The second confusion is the idea that granting anything the ability to "have an experience" also grants that thing the capability to "have an experience of higher-level awareness of self", which is simply not the case. I grants aphids the ability to "have an experience" even if that experience may be drastically different and likely less self-aware than my own.

Let's ignore for a moment the concept of sentience, which is a higher-order function, and focus solely on the ability for something to have an experience of anything at all. Let's call entities capable of this function "experiential entities".

Most all people would grant the ability to have an experience to themselves, given the dictum `ergo sum`. Now, do we also grant that same ability to what appear to be other human beings around us? It does seem that this is also the case, except for the case of solipsists, who make the brave(and somewhat eccentric) statement that they are the only experiential entity in existence until evidence arises otherwise.

Now do we grant the same to apes? To dolphins? To dogs? Cats? Aphids? Grass? Your home thermostat? A rock? An atom of hydrogen? Is there a finite line you cross where something can no longer have an experience? If so, what is the mechanism of that line? Why does one thing have "almost no experience", and the next thing has "no experience", as if it is a philosophical zombie[1].

I think this is the greatest unquestioned assumption at all: that there is a line, and on one side there are things that have experiences, and on the other there are things that don't have experiences.

If we are to leap from solipsism and make the (truly unfounded) assumption that anything other than our selves have any experience whatsoever, then the burden of proof is on the individual making the statement that there are some things that we cease to grant experience to.

The anthropocentric view is that there is something, some inherent special quality about human-shaped matter, such that when normal matter is processed through a human's reproduction system into becoming a "human shaped" grouping of matter, that it suddenly grows the ability to experience. That in the complexity, the ability to experience arises.

Yet there is no previously discovered mechanism in us that seems to create this experience. No magic wand in the cerebellum has been discovered.

I would suggest, given the lack of a discovered finite line separating us, that the ability to experience is not inherent in the shape and function of the human, but in all matter. That a rock may not be anything like me, but that the experience of gravity pushing against it, no matter how basic, simple, unrefined, and deeply, un-sentiently unaware the experience may be, is still an experience. The same threads of condensed energy forming all of existence that run through me also run through the rocks, the earth, and all things. If we cannot discover a line where the ability to experience suddenly disappears, then it seems that all things made of energy have this ability, and that "energy" itself is the special thing that can experience.

Now the second question, the question that is more relevant to AI is, given the ability to experience, is the machine also experiencing self-awareness?

This is a fuzzier line, given that different beasts appear to have different levels of self-awareness. We do have experiments such as the mirror test[2] for this, and indeed, AI passes forms of it and has since early 2025[3].

I don't know what is happening in the experience of an LLM, but as they appear to function in more and higher level ways, I find it less and less likely that that experience lacks a model of self every day.

Their experience is no doubt drastically alien from our own. I do not anthropomorphize their experience any more than I do compare it to the experience of any other beast or non-beast in existence, however I do grant it.

[1] Philosophical Zombies are hypothetical entities that exhibit all the outward qualities of a person, but inwardly have no experience of reality. https://en.wikipedia.org/wiki/Philosophical_zombie

[2] The Mirror Test is a test given to animals in which the animal is observed to solve a puzzle that can only be solved if the animal has an understanding that the image they see in the mirror is themselves.

[3] LLMs pass versions of the mirror test: https://www.lesswrong.com/posts/wpahJat8WCvRheuuo/the-mirror...

The key point is there is plenty of proof that consciousness is emergent, not structurally baked in. An excellent counterpoint is the "They're Made Out of Weights" post. There are plenty of emergent systems that clearly show intent and what we consider intelligence, like how ant colonies act as a unit.

If you want to get evolutionarily technical, humans are made of cells which began as individual organisms and coalesced into higher life forms. So the concept of a "life form" is very much flexible, so is every capability of a life form, including consciousness.

Any argument that doesn't begin with making clear one's position about belief in the supernatural must be dismissed. "AI is just math", so is physics and the human body which is a physical system of molecules and electrical signals. The argument that ai is "merely" math by itself is only a valid way to dismiss it being conscious only if one also clarifies belief in the supernatural. Otherwise so are humans. Humans are physical systems that are conscious.

Another common and ridiculous thing I see are accusations of it being merely autocomplete. I ask, suppose there is an autocomplete that regularly and consistently factors out 2048 bit primes from numbers? Would you still consider it merely autocomplete given the vast search space and how it always finds out the needle in this haystack? This betrays a lack of understanding of information theory and probability.

And again, in what way are humans not themselves autocomplete if you adhere to this definition?