Ted Chiang doesn't understand consciousness. To be fair, nobody really does, so the only forced error is acting like you DO understand consciousness.
Artificial intelligence is not conscious – Ted Chiang
361–370 of 1001 posts
Re: Artificial intelligence is not conscious – Ted Chiang
#362The one thing we know FOR CERTAIN is that having actually read the article is not a prerequisite for consciousness.
The number of people posing questions that are already directly addressed in TFA is impressive.
Re: Artificial intelligence is not conscious – Ted Chiang
#363I've thoroughly enjoyed the couple of short stories of his I've read, so this was a highly disappointing read.
Firstly, many of the technical arguments are of the "stochastic parrots" variety, which almost nobody really believes anymore. Ironically Anthropic's own research shows pretty abstract, conceptual things happening in the weights (cf Golden Gate Claude.)
Secondly, this seems to improperly mingle consciousness, intelligence and morality. Consciousness is not required for either of the other two. As TFA itself says, the model's "morality" is some aggregate function of the morality encoded in its training data... but that means it does exist and does influence its outputs! Even if it's not conscious there have been -- and will be -- a zillion times where the models must make choices that have moral implications. We know the models got many of them so wrong, which indicates the need for some mechanism to ensure the model is "aligned" with what somebody considers "good", which for Claude is the constitution.
Now, Anthropic does seem to go overboard with Claude's "well being" but that does not mean there are some very practical reasons to be concerned about that: LLMs behave like humans because that's what their training data contains, and humans lash out when their well-being is threatened, so why would LLMs not do the same?
I think the core problem is that the author has an extremely anthropocentric view of things. Here's an interesting rabbit-hole to go down: some researchers believe plants feel pain. (How's that for a plot twist, vegans?) The consensus is against them, but their counter is that we have a very human-centric definition of pain. The fact remains that plants show a number of responses signifying distress analogous to animals in pain including taking defensive actions and warning their neighbors.
We don't think that qualifies as "real" pain, but that doesn't make it any less real for the plants!
Similarly if an LLM believes it is in "pain", we know it's not real... but that doesn't make it any less real for the LLM either. And concerningly, it has far more degrees of freedom to react. (Who knows when somebody will hook up an MCP to our nukes.)
> I believe creating software that is conscious and deserving of moral consideration will be so difficult that we’re unlikely to do it accidentally...
I actually reach the opposite conclusion: It is so impossibly difficult that our limited primate brains could only ever do so accidentally. Did some distant ancestor of ours intentionally make fire... or accidentally discover it?
Re: Artificial intelligence is not conscious – Ted Chiang
#364Earlier quoted context omitted.
It can't even natively understand how many letters there are in words - how will it understand the meaning?
I wish people would do even the most basic amount of research into LLMs before opining about what they can or cannot do. There are very principled reasons why LLMs do not know how many letters are in words, and it says nothing about their facility for understanding meaning. Tokens are the most basic input unit of an LLM. But tokens don't generally correspond to words or letters, rather sub-word sequences. So Strawber…
>The widespread mistake people keep making is assuming the development of intelligence in LLMs should follow the same trajectory that human intelligence takes as it develops into adult levels of intelligence.
To be fair, almost everyone who claims LLMs are conscious tends to claim that they are conscious in exactly the way that humans are, to the point of stating that human brains are also just complex next-token prediction machines with a random seed. It's basically religious arguments on both sides.
Re: Artificial intelligence is not conscious – Ted Chiang
#365It's becoming painfully evident that no one really understood the argument behind the Turing test.
If nobody understood the argument, then it was either weak or unclearly communicated.
Heh. And yet it's widely respected.
How odd the crowd is here tonight. Very aggressively disagreeable.
Re: Artificial intelligence is not conscious – Ted Chiang
#366The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness. It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember…
> It's not changed by the experience
The entire file is not changed, but the KV cache is.
> It doesn't remember anything
The model definitely remembers previous exchanges within the same conversation.
Re: Artificial intelligence is not conscious – Ted Chiang
#367The analogy I make is between airplanes and birds. Birds are alive, are conscious, flap their wings, and fly. Planes are not alive, are not conscious, do not flap their wings - and fly. Similarly, current AIs are not alive, are not conscious - but think. All prior entities that thought, were human, so the only experience humans had with other thinking entities were other humans. The huge mistake now being made is ass…
What if somebody simulated all neurons of a bird and fed them appropriate stimuli? Would a bird neural replica be conscious? It would flap, that's for sure.
This has been tried with much simpler organisms, it did not behave like the real thing thus far. There was a paper about it, there now seems to be a project to push on the frontier
Re: Artificial intelligence is not conscious – Ted Chiang
#368> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation Regardless of bigger issues, this kind of statement reveals a deep misunderstanding. Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power. If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is…
I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism. So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably…
I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle.
sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.
Re: Artificial intelligence is not conscious – Ted Chiang
#369The fact that a LLM is essentially immutable would be my biggest argument against consciousness or self-awareness. It's a big file with a bunch of coordinates describing spatial relationships between tokens. When you give it a prompt, it uses those relationships to generate a string of tokens that is a statistically likely response to that prompt, then it stops. It's not changed by the experience. It doesn't remember…
Reinforcement learning changes the model. So it can and does change and remember based on experience. Eventually reinforcement learning can happen in real time.
But is the model aware of the training? Unless you hook the model up to an MCP server, or something similar, and have it analyze the RL changes, it will not know if it has changed or not. Even if it is real-time RL, it is not aware of the previous state.
Re: Artificial intelligence is not conscious – Ted Chiang
#370I think about Star Trek: TNG’s “Measure of a Man” a lot lately. We can be so confident to decide what is and isn’t alive from vibes alone. The conclusion I’m currently at is that I don’t know and probably can’t ever know. Maybe you’re all philosophical zombies. Maybe I am one too! But at some point we will get close enough that it hopefully becomes obvious that we must tread carefully. The entire episode is incredibl…
I think about this from the other end. It cannot be considered a conscious being. There just isn't a world in which we should start to think of a machine using ethics we reserve for humans. AI is essentially infinitely reproducible at zero cost, and won't suffer from decay etc. There's not scarcity to preserve. So, I'd turn off an AI in a moment to save property or real possessions or money. I'd sacrifice property an…
Consider that it is very costly to train a model. It is only cheap to copy because the substrate we instantiate it onto is a substrate we have designed to be fully readable.
I think you should reconsider this viewpoint. Suppose that we really can create silicon-based consciousness, in that case your view would result in a huge amount of suffering.
Take some other basis for dismissing digital consciousness, this one is too dangerous.