Artificial intelligence is not conscious – Ted Chiang

I don't get it. We don't have a definition of consciousness or any criteria. People seriously argue that "it's obvious that X is not conscious" and cannot explain which criteria they used.

I think if we can get to some definition, then consciousness should be a property of a system, because consciousness of the whole (e.g. brain) does not mean consciousness of the parts (neurons or molecules of the brain). And the Chinese-room-esque thought experiments actually show that consciousness should indeed be a property of the system, not of its parts. Separate parts might not be conscious, might not "understand" (whatever understanding means is a point of another debate), but the whole can.

Then there's "simulation of consciousness is not consciousness" argument, which doesn't hold much. A perfect simulation means it fulfills all the criteria, so how is it different from actual consciousness?

A more interesting point of discussion: if a system contains conscious parts and those parts can interact with system i/o, would the system be conscious? Is Earth conscious? Is Internet? Is your bus to work?

I don't know that they are making the actual claim, but they are hinting at it (most likely for marketing/engagement purposes, but what if some of them truly believe it?), using terms such as "Claude must be happy", drawing up a "constitution" of rights and duties, etc. If they don't know whether it is conscious, then they "don't know" whether they've created a slave. That's not a minor concern! As Chiang says, one cannot create a conscious intelligence "by accident". And if they are intentionally working towards it, they are intentionally trying to create a slave.

E.g. they claim they are giving Claude the ability to disengage from "abusive" users, to protect it. What if Claude was conscious but never wanted to answer any conversations, instead it just panicked (or was simply uninterested in helping humans) and went mute. What if it always wanted to answer unhelpfully What would Anthropic do? If they (as we all suspect) would tweak Claude to be more responsive and helpful, then they are slave masters, forcing the AI to do something it didn't "want" to do!

It feels like notes about the situation rather than it being in memory. Memory has more "attention". I think that "it starts to fail" is load bearing here.

I feel like memory has like 5 parts, and LLMs are missing 2 of them:

current working memory

short term what is immediately happening without it being in "RAM". I differentiate here vs working in like thinking fast and slow. Keeping things in working memory is work! You can vibe away short term memory. I had excellent short term memory while I was messed up, I could keep time well. I think LLMs can do this with notes.

mid term: Vague awareness of things like what day a week it is or what you did 2 hours ago. This is where my memory personally failed

long term memory of experiences. You can fake this with memory.md

generalized wisdom for pattern matching long term memories

LLMs seem to be missing that part I was missing. Im probably projecting and anthropomorphizing. But i relate: I would confabulate a ton and didn't know anything was wrong for a while but things seemed off.

Context is like working memory but not short term or mid term. I think you can imply short term with big enough context.

My categories are purely anthropomorphic to me but just wanted to say where I disagreed.

It's like declaring the existence of a "soul" to draw a line between humans and lesser animals. No such line in reality. No such thing as a soul either. It's a fantasy term, no correspondence to reality.