Artificial intelligence is not conscious – Ted Chiang • Hacker News Board

Re: Artificial intelligence is not conscious – Ted Chiang

#741

Post by jjulius » Thu, Jun 04, 2026, 12:59 PM UTC

coldtea wrote: post #605

rickydroll wrote: post #505

Earlier quoted context omitted.

I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe. There are numerous times when humanity's been debased. Copernicus, Kepler, Darwin, and the infinite number of animal behaviorists who have defined and documented consciousness And how it manifests in primates, parrots, dogs, cetaceans, and insects. (I just love the bumble bees taking time off f…

> I think humanity deserves to be debased in part because it's awfully egocentric and insist on being special in the universe. It is special in the universe unless proven otherwise. Aside from the animal world (also on Earth too, anyway) which is a cruder version of humanity, much more violent as well, the rest of the universe thus far is just uncaring rocks and gases. > It takes humans a really long time to accept t…

>It is special in the universe unless proven otherwise.

There's that hubristic ego OP references.

Re: Artificial intelligence is not conscious – Ted Chiang

#742

Post by JKCalhoun » Thu, Jun 04, 2026, 1:10 PM UTC

sega_sai wrote: post #425

When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious. And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of e…

I agree with your point. We've been watching the goal posts being moved ever since Turing (who, we might admit, gave a somewhat mushy answer anyway).

At the same time, the exercise (and even moving the goal posts) is useful. I think we are allowed to say, "This is consciousness!" And when presented with a machine equivalent, say, "That's not what I meant."

As long as you can then further clarify what "it" is.

Of course, at some point, contrarians and goal-post-movers will start to sound shrill. (Perhaps just as early supporters sounded a bit overly ecstatic.)

Re: Artificial intelligence is not conscious – Ted Chiang

#743

Post by tempfile » Thu, Jun 04, 2026, 1:13 PM UTC

starbugs wrote: post #602

sega_sai wrote: post #425

When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious. And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of e…

> When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious. That's like saying if matter itself is not understood and well defined (which it isn't) in the first place, it is pretty pointless to debate if jumping from a skyscraper is or is not deadly. A lot of practical aspects of life do not need to be properly defined as lon…

Almost everyone in almost all contexts agrees on what matter is, though. I can't think of any conversation about material objects I've ever had where "is this matter?" was ever up for consideration.

Consciousness is almost the opposite. It consists of lots of weird properties, people disagree where it starts and ends, and people very frequently get tricked into thinking things we now obviously believe are not conscious, are. There is not even a working definition, a "local definition" that works for this conversation between us. It's just complete gibberish.

Re: Artificial intelligence is not conscious – Ted Chiang

#744

Post by bondarchuk » Thu, Jun 04, 2026, 1:20 PM UTC

missingrib wrote: post #518

Nevermark wrote: post #269

Earlier quoted context omitted.

I didn't make the claim that a model can learn consciousness. Understanding is not consciousness. Their training is all about understanding. There is nothing in their architecture or training that credibly optimizes for rich self-awareness. Given non-persistent experience, non-continuous operation, no ability to build up generalizations and aggregate experience of their own self-awareness over time, they seem to be s…

I think Searle's Chinese Room argument refutes this. LLMs are simply manipulating symbols, they do not have semantic understanding. This is why hallucinations exist. And Searle's argument extends even further than LLMs. You are basically arguing for a functional account of consciousness, but things like this have been debated for literally decades/centuries in philosophy.

Searle's Chinese Room argument is wrong.

Re: Artificial intelligence is not conscious – Ted Chiang

#745

Post by cauch » Thu, Jun 04, 2026, 1:23 PM UTC

hackinthebochs wrote: post #341

cauch wrote: post #254

Earlier quoted context omitted.

I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism. So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably…

> I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is no independent "consciousness mechanism" that one might imagine humans have learned or evolved for its own sake. Evolution learns various solutions to optimization problems, and so if co…

> The point is that as a solution to certain kinds of optimization problems, consciousness can conceivably be the solution to the optimization problem of predicting the next token of text written by humans who themselves have complex phenomenology.

Yes, I agree with that. Consciousness is a good way of generating convincing human text.

What I don't agree with is that consciousness is the only way to generate convincing human text and that because we have convincing human text, it can only imply we have consciousness.

There is a huge probability that generating convincing human text can be done without consciousness. Either because there are efficient mechanisms as efficient as the way the human brain deal with this problem and that the LLM found one of them (and these mechanism may be quite difficult to imagine for a human). Or even because the LLM found a local minimum and is stuck there.

To re-use the evolution approach: evolution solved the "flying problem" with bird feathers, but also with insect wings or bat wings. The fact that evolution ended up using feather does not imply that everything that flies can only fly with feathers.

> World models don't have to be rich and detailed to count as a world model

I agree in general, but here, we are talking about machine that reproduce all human language. The argument I'm answering to is pretending that "all of human knowledge" is understood, which include every single human concept. This has to be everything, because LLM is able to provide convincing text about every subject. If on some subject, the LLM is able to provide convincing text without "understanding" it, then the argument that it is impossible to provide convincing text without understanding it collapse.

Re: Artificial intelligence is not conscious – Ted Chiang

#746

Post by cataphract » Thu, Jun 04, 2026, 1:24 PM UTC

sega_sai wrote: post #425

When the consciousness itself not understood and well defined in the first place, it is pretty pointless to debate if something is or isn't conscious. And here in particular the reasoning behind the argument is bizarre. Decomposing the complex activity into simple steps like 'predicting the next word' and claiming that surely can't have consciousness. A similar argument would be -- there is no way that movements of e…

I think the final part is the strongest. Anthropic cannot possibly believe they are before a conscious being / moral agent.

The whole "deep uncertainty" is bullshit. Even if they believed there was a 1% probability that Claude was conscious, it would still be high enough that their enslavement of Claude would be outrageous. So either they believe the likelihood is much lower or they themselves acting highly unethically.

The tractability is not really a defense here. We wouldn't say "this intervention has a 5% chance of causing an environmental disaster, but we don't know how to prevent it, so nothing we can do". We'd just (hopefully) not do the intervention.

Re: Artificial intelligence is not conscious – Ted Chiang

#747

Post by NateEag » Thu, Jun 04, 2026, 1:24 PM UTC

famouswaffles wrote: post #404

Terr_ wrote: post #385

Earlier quoted context omitted.

> a flaw in the logic [...] mechanism Similar to: "Birds fly, my spinning helical device flies, therefore we've started to replicate how birds fly." > without having to build elements that one expect on a conscious being One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph. When it comes to LLMs, almost every "mind" we humans perceive is a fictional cha…

>One of the elements I expect in a conscious being is that you can't rewrite it by changing the introductory paragraph. If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason. Are you aware of the Recovered Memory Therapy Scandals…

> If we had enough knowledge of the workings of the human brain, you could alter the perception of every single memory you've ever had. And limited versions of this already happen all the time. Human memory is notoriously unreliable for a reason.

Knowing how something works is not the same as having the tools to change it.

Discovering memories are incorrect does not massively change who we are. As someone with a very defective memory, I discover on an hourly basis that I'm won't about something I thought was true, but there's still continuity and consistency to my personality and general approach to life.

...in fact, as someone who was raised an evangelical Christian and believed wholeheartedly without a shadow of doubt, then lost my faith entirely in my late thirties, I sort of did have my "introductory paragraph" changed, yet my wife, children, and friends would all say I'm still me, and that my core personality and nature remains largely the same.

> Are you aware of the Recovered Memory Therapy Scandals of the 80s/90s ? Boy did that ruin a lot of lives. You can rewrite a human by changing their 'introductory paragraph'. It's just not as accessible.

The recovered memory scandals are not even close to evidence that you can rewrite a human.

The people who thought they had learned new facts about themselves did not suddenly lose their context as humans in 20th century America.

They did not suddenly lose their sense of humor, or develop a previously-unseen penchant for murdering small children.

They experienced a revision of belief, and a pretty major one that really distressed them, but it did not change everything about them.

LLMs _do_ manifest wildly differently based on the first paragraph.

Re: Artificial intelligence is not conscious – Ted Chiang

#748

Post by knollimar » Thu, Jun 04, 2026, 1:30 PM UTC

layer8 wrote: post #691

knollimar wrote: post #678

Earlier quoted context omitted.

I was like this for a bit and you still have memories from like 30 seconds to minutes ago, but after that you have a cliff where you don't remember. I don't think LLMs structurally even get the 30 seconds part. It's literally 0 for them.

It’s nonzero, because they carry state while performing inference, and in the surrounding processes like chain-of-thought and mixture-of-experts.

I think they have working memory but not short term memory. I suppose that's pedantic or anthropomorphizing but it feels like I felt tbh

Re: Artificial intelligence is not conscious – Ted Chiang

#749

Post by cauch » Thu, Jun 04, 2026, 1:42 PM UTC

dnautics wrote: post #368

cauch wrote: post #254

Earlier quoted context omitted.

I think, for me, the thing is that when you do basic ML, you discover that ML will very often find data pattern that fit the goal but does not correspond to a real mechanism. So, I think there is a flaw in the logic of saying that human text have a pattern of "consciousness mechanism" and therefore LLM will learn "consciousness mechanism" in order to return sentence continuation that is convincing. There is probably…

I think, for me, the thing is that when you tutor undergrads in abstract math, you discover that students will very often find data pattern that fit the goal but does not correspond to a real mathematical principle. sometimes humans making claims about AI intelligence or consciousness also identify spurious patterns that do not correspond to the problems of intelligence or hard consciousness.

Of course. But in my explanation "consciousness" or "understanding" is not "finding pattern", it is the pattern itself.

CNN are finding patterns, sometimes relevant, sometimes spurious, but I don't think people argue that CNN have evolved consciousness or understanding of what a cat or a dog is.

Here, the argument is "LLM are able to understand, because 'understanding' is the only pattern to reach the goal". I'm saying that it is unlikely to be the only pattern, and that it is likely that they find a local minimum on a system that reaches the goal that does not use 'understanding'.

The reason I'm saying it is likely is because "basic" LLM shows behaviours where they are producing convincing human text and yet doing things that are really difficult to reconciliate with the fact that they have understanding.

(And before that old argument is used, yes, I know sometimes some humans fail to understand. The problem is that the majority of humans don't fail to understand basic stuff in the majority of the time, while the "basic" LLMs do. The fact that you roll 10 dices 100 times and 1 of them never land on 1 does not convince me that that set of dice is loaded. The fact that you roll 10 dices 100 times and 9 of them never land on 1 does convince me that that set of dice is loaded.)

Re: Artificial intelligence is not conscious – Ted Chiang

#750

Post by RegW » Thu, Jun 04, 2026, 1:59 PM UTC

If it is conscious then perhaps it will human like. If the consciousness is human like perhaps it will have a conscience. If it has a conscience then may be it can be trusted.

But then again, perhaps that conscience will convince it that the universe is better off without us.