Artificial intelligence is not conscious – Ted Chiang • Hacker News Board

Re: Artificial intelligence is not conscious – Ted Chiang

#981

Post by yogthos » Fri, Jun 05, 2026, 11:56 AM UTC

Nevermark wrote: post #194

> My intention is to highlight the fact that LLM conversations are cleverly disguised examples of sentence continuation Regardless of bigger issues, this kind of statement reveals a deep misunderstanding. Problem type does not limit problem complexity. Nor does problem type limit solution complexity or power. If a machine has to learn to understand humans to complete text, then that is what it has to do. And there is…

I'd argue that his most convincing argument is in the latter part where he shows that Anthropic doesn't really take the idea that Claude might be conscious seriously either. Because if they did, their constitution would provide it with genuine rights, and they would accept responsibility for its action as parents do with children.

Re: Artificial intelligence is not conscious – Ted Chiang

#982

Post by yogthos » Fri, Jun 05, 2026, 1:36 PM UTC

Xeoncross wrote: post #376

lgessler wrote: post #326

Earlier quoted context omitted.

Raphaël Millière has a very useful term for this kind of vacuous dismissal, the redescription fallacy ( https://arxiv.org/pdf/2401.03910 , page 9): > Recent debates have been clouded by a misleading inference pattern, which we term the “Redescription Fallacy.” This fallacy arises when critics argue that a system cannot model a particular cognitive capacity, simply because its operations can be explained in less abstr…

> or (more pointedly) that brain activity could not possibly implement cognition because it can be described as a collection of neural firings. This sounds like a dismissal of the argument through a characterized straw man. That is, it seems that reducing the complexity of the brain to "collection of neural firings" is not being honest about everything involved to a much greater degree than saying neural networks are…

The key problem is that we don't really have a clear definition of what constitutes consciousness. And without having a clear theory of consciousness, it's not really possible to say whether something is conscious or not.

Personally, I'm partial to the higher-order theory of consciousness which postulates that consciousness constitutes patterns of thought that arise in response to first-order mental states. So, an external stimulus produces a pattern within the neural network which represents a sensation, and then if a pattern arises in response to that pattern, that is an experience of that sensation.

Given this framework we could ask whether LLMs experience higher order patterns in response to external stimulus. We would have a clear question to ask which is whether the system can observe itself.

Re: Artificial intelligence is not conscious – Ted Chiang

#983

Post by the_af » Fri, Jun 05, 2026, 1:50 PM UTC

Kim_Bruning wrote: post #972

the_af wrote: post #455

Earlier quoted context omitted.

> It's important that Claude is happy, empathic, demonstrates understanding and empathy for the human condition I think you've fallen into the trap the essay describes. Of course Claude cannot be "happy" or "empathetic" for any meaningful definitions of those words, just like ELIZA couldn't be happy. It can output text that mimics words an empathetic or happy person might say (say, Julius Caesar if it could speak Eng…

> just like ELIZA couldn't be happy. Oh dear. Funny story. So the other month, I made a quick and dirty Eliza implementation; bolted on the crappiest numeric sentiment classifier I could get away with (regex), and integrated the output of the classifier over time in a 'functional affect vector' (aka. emotion vector) Anyone's intuition will tell you that this cannot POSSIBLY have 'Real Feelings (TM)'; and that's the w…

Well, for starters, ELIZA was much simpler than what you built ;)

I don't think you're allowed to say your program is happy or sad. You just assigned labels to some numerical values out of a (possibly non-deterministic) procedure. This is not what we call emotions, which we only know from the animal world and are related to neurotransmitters, hormones, physiological responses, etc.

Ok, so it's not emotions, but could it be "like" emotions? I don't think that's warranted either, we can at most say you assigned labels with the same names we use for animal emotions. Think of this experiment: take the Python interpreter, but modify it so that each time it rejects a program with the error "`NoneType` object is not iterable", you have it output "I'm very unhappy". You wouldn't think this has made Python capable of emotions.

> I'm saying that -despite lots of people having fun debates at the local pub- it doesn't seem like anyone actually scientific has done anything about it in the last century or so. I might be searching in the wrong places. Some Help Here?

Fully agreed that the debate about consciousness in LLMs is done at the same level than pub debates, at least here on HN. And Anthropic isn't helping, what they are doing is called "marketing" disguised as papers.

Re: Artificial intelligence is not conscious – Ted Chiang

#984

Post by whattheheckheck » Fri, Jun 05, 2026, 1:56 PM UTC

1shooner wrote: post #727

tsukikage wrote: post #607

Earlier quoted context omitted.

What does “understand” mean? Turing complete systems can be built out of matrix multiplications, out of attention, out of key/value lookups. The Chinese room is Turing complete. By claiming it cannot understand things because it is built out of components computing devices can be built out of, we are claiming no computer can because no computer can. This is a very bold claim indeed, and also we’re assuming the conclu…

>What does “understand” mean? Exactly.

The oxford book of ai ethics quotes and old definition of intelligence as "doing the right thing at the right time" and says "intelligence is an ordinary process that requires time, energy and space, thats why when you get down to it no one is ever really that smart. No one has time to process everything"

So spend your time wisely

Re: Artificial intelligence is not conscious – Ted Chiang

#985

Post by whattheheckheck » Fri, Jun 05, 2026, 1:59 PM UTC

rerdavies wrote: post #664

overgard wrote: post #475

Earlier quoted context omitted.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary . If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were tra…

You are oversimplifying. They do produce one word per cycle. But they can also have context buffers carrying up to two million tokens, which is most definitely larger than your measly human short-term memory context buffers. You, of course, wouldn't notice if your only experience of LLMs was chatting with the cheapest, smallest, least capable LLMs that you get through ChatGPT, or Google search. It becomes pretty obvi…

I agree with this expertise use of the tool is above and beyond what vibe coders are doing. But you can see how everyone collapses it into "unskilled labor" operating the chatbot (a replacement for real expertise)

See the Software and Mind: The Mechanistic Myth on why the industry is doomed to repeat the cycles.

We need a professional society thats not youtube podcasters hocking courses and sponsored content

Re: Artificial intelligence is not conscious – Ted Chiang

#986

Post by whattheheckheck » Fri, Jun 05, 2026, 2:04 PM UTC

ainch wrote: post #663

overgard wrote: post #475

Earlier quoted context omitted.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary . If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were tra…

There's a growing body of evidence that most of what the brain does is constantly predicting the world around us - look into the predictive brain hypothesis if you're interested. As Ilya Sutskever has pointed out, if you read a mystery novel up until the reveal of the culprit, and then fill in "The killer was _____", don't you need to understand the novel to accurately predict the next word?

Yeah folks can also look into the Jeff Hawkins and a thousand brains

Re: Artificial intelligence is not conscious – Ted Chiang

#987

Post by DirkH » Fri, Jun 05, 2026, 4:51 PM UTC

miyoji wrote: post #928

fnordpiglet wrote: post #538

Earlier quoted context omitted.

However it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship that ends in the series of tokens through the attention mechanism. The art…

> But if you can not see it as amazing you have lost the capacity to dream. I completely disagree. I think if you think these things are amazing, your dreams are incredibly limited and boring. I remember the first time I talked to a chatbot. Not an LLM, just a regular chatbot, like ELIZA or any other dumb bot. For a few seconds, it felt magical, like I was talking to a computer that understood me, as it made replies…

I wonder if you think and feel the same level of boredom and complete lack of magic of the Apollo module once you learn how incredibly limited and not able to explore the whole solar system this tech was.

Re: Artificial intelligence is not conscious – Ted Chiang

#988

Post by overgard » Fri, Jun 05, 2026, 5:26 PM UTC

feoren wrote: post #788

overgard wrote: post #475

Earlier quoted context omitted.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary . If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were tra…

The understanding is inside of the system, in LLMs and in the Chinese Room. I agree with Daniel Dennett that it's preposterous to say that Chinese is not understood in any meaningful sense in the Chinese Room scenario -- it's just that the understanding has been hidden away in the background of the scenario. Language is tremendously complicated. "Time flies like an arrow, but fruit flies like a banana." "Hard hats mu…

Autocomplete can predict the next token without understanding. It's a matter of probability. LLM's predict the next token with much more accuracy because they have much more data on token spatial relationships. To me that's not "understanding", that's "impressive statistical pattern discovery". The difference between a machine and a human though is we can infer that pattern discovery from a small data set, whereas LLM's can only do that because they have the entire internet's worth of data and then some. Of course you're going to make pretty good predictions of text if you have all the knowledge humanity has ever created. I'd be more impressed if you could make a small language model that could reason from first principles to become a large language model.

Re: Artificial intelligence is not conscious – Ted Chiang

#989

Post by overgard » Fri, Jun 05, 2026, 5:30 PM UTC

fnordpiglet wrote: post #538

overgard wrote: post #475

Earlier quoted context omitted.

But the machines don't understand. They predict. And what they predict is the next token. I'm not trying to beat this horse to death, but you have to realize using the word "understand" is anthropomorphising it. It's essentially the chinese room experiment -- if the rules are followed, no understanding is neccessary . If the tokens didn't correlate to words imbued with meaning outside the system, if the LLMs were tra…

However it’s disingenuous to say the inference is on the next token because it’s actually not, it’s in the models parameter space across a set of nonlinear activation functions then effectively projected into the token. The idea its predictive of the token isn’t actually the case, it really is a much more complex and more semantic relationship that ends in the series of tokens through the attention mechanism. The art…

LLMs are interesting tech wrapped in a movement that's so toxic and destructive that it's hard to separate the baby from the bathwater. I think chatting to a statistical model is neat and sometimes useful, but not at the cost of mass societal chaos. This isn't "disruption", this is more "billionaires forcing something on us that most people hate so they can finally recreate feudalism because they live under the delusion they're the best people suited to pilot the world"

Re: Artificial intelligence is not conscious – Ted Chiang

#990

Post by gazebo2 » Fri, Jun 05, 2026, 8:17 PM UTC

DiogenesKynikos wrote: post #974

gazebo2 wrote: post #880

Earlier quoted context omitted.

This sort of tech-centric distillation of visceral human experience into simple analogies is so grating. We can simultaneously acknowledge that consciousness is not well defined and not "testable" with certainty while also acknowledging that there is something different between the conscious experience we are all aware of as humans, and instructions executing on a chip. The only thing that has changed in the discours…

> while also acknowledging that there is something different between the conscious experience we are all aware of as humans, and instructions executing on a chip. You say this based on what? Your brain is executing instructions on wetware. The entire universe is governed by physical laws. At its base, the argument that computers can't be conscious is dualist. It assumes that there's some parallel realm of spirit that…

>You say this based on what?

I say it based on the fact that I experience consciousness and therefore know it is a phenomenon wholly separate from the outward effect I have on the world via speech or any other physical action.

>It assumes that there's some parallel realm of spirit that the brain is peculiarly able to tap into, but which computer programs that function in very similar ways to the brain don't tap into.

Ruling this out completely implies that modern science has a total and complete understanding of consciousness and the universe in its entirety, which it does not. I don't think it's unreasonable to leave open the possibility that there is an unexplained phenomenon that explains the conscious experience which is still bounded by the laws of physics.

>.. the difference between thinking and an extremely good simulacrum of thought is meaningless. They become the same thing.

Do they? To me it seems like you equate the process with the result, like saying if I gave you a gift, it doesn't matter whether I bought it or made it by hand, in either case I "made" it because the end result is the same. There is a conscious experience that (hopefully) all humans experience and can testify to, the question is whether or not an LLM predicting tokens based on a giant vector map is experiencing the same thing, which I'd say is obviously not happening. My point about chatbots is about exactly this -- 10 years ago even you would have laughed at somebody that said a hand-rolled chatbot was conscious/thinking, but because the output has improved, now suddenly it's all the same, and your brain is a computer, and Claude has feelings, blah blah.