Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs and self-referentiality

scottaaronson.blog

Post reply on HN
Print view View original

81–90 of 95 posts

Re: LLMs and self-referentiality

#81

Post by GPerson »

andai wrote: post #56
thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition. Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.

You should read his later work called I Am a Strange Loop instead of GEB, as this is the author’s preference.

Re: LLMs and self-referentiality

#82

Post by nextaccountic »

k__ wrote: post #42
thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

Any signal that allows the model to see what's wrong helps. Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct. Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.

> Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.

A global database of facts would make easier for AI to stay factual

Also ironically it would also make it easier to align AI to do things like consistently censor or distort some political facts

Re: LLMs and self-referentiality

#83

Post by nextaccountic »

gwd wrote: post #50

Two comments on this, trying to take a "which hypothesis fits the evidence" approach. First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what i…

> First, an LLM describing its own experience is not actually proof that it has any experience to be aware of,

Likewise, humans describing their own experience is not actually proof that they have any experience to be aware of

Re: LLMs and self-referentiality

#84

Post by nextaccountic »

jwr wrote: post #20

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious? We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really de…

I think this is exactly the same problem to decide whether any human is conscious, just because they say they are conscious. Humans can act as if they are conscious too, they can even lie to themselves, their whole life, and never discover the truth about their own mind

Re: LLMs and self-referentiality

#85

Post by alchemism »

GPerson wrote: post #81
andai wrote: post #56

Earlier quoted context omitted.

Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition. Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.

You should read his later work called I Am a Strange Loop instead of GEB, as this is the author’s preference.

I think of this book often in the LLM age.

Re: LLMs and self-referentiality

#86

Post by gwd »

nextaccountic wrote: post #83
gwd wrote: post #50

Two comments on this, trying to take a "which hypothesis fits the evidence" approach. First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what i…

> First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, Likewise, humans describing their own experience is not actually proof that they have any experience to be aware of

Indeed, but I actually kind of mis-spoke here. The question is less about having an experience to be aware of, but the ability to accurately reflect internal state.

Even humans need to learn how to read their own internal state (e.g., saying "I got mad" rather than "I felt ashamed because I wasn't living up to my picture of what a good person is, and covered up the shame with anger").

But if an LLM were to say, "I'm sad" or "I'm happy", does that actually correlate to anything? I'm OK with saying "The LLM was sad", if there is an internal state that leads to observable changes in behavior correlating with the kinds of changes in behavior humans have when they're sad. The question is, if the LLM says "I'm sad", is that because it has that internal state (self-reflection)? Or is it because that's the kind of thing a human would say in that context?

I think both are possible. I also think that between internal probes and behavioral testing, it should be possible to determine which one is closer to the truth. I'm just pointing out that "LLMs talk about their internal state" isn't proof that LLMs have self-referentiality, without additional evidence that the talk is actually related to their internal state.

Re: LLMs and self-referentiality

#87

Post by andai »

GPerson wrote: post #81
andai wrote: post #56

Earlier quoted context omitted.

Not sure self-reference solves the metacognition thing; an ant can pass the mirror test but probably lacks metacognition. Though I haven't read GEB so I'm not sure how the strange loop thing ties in with either of those.

You should read his later work called I Am a Strange Loop instead of GEB, as this is the author’s preference.

Thanks. I actually read this book in high school. I remember being very impressed with it and bringing it to school.

I don't remember a single thing! (Which is unusual for me, I usually remember much of what I read.)

I shall have to read it again :)

Re: LLMs and self-referentiality

#88

Post by nextaccountic »

gwd wrote: post #86
nextaccountic wrote: post #83

Earlier quoted context omitted.

> First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, Likewise, humans describing their own experience is not actually proof that they have any experience to be aware of

Indeed, but I actually kind of mis-spoke here. The question is less about having an experience to be aware of, but the ability to accurately reflect internal state. Even humans need to learn how to read their own internal state (e.g., saying "I got mad" rather than "I felt ashamed because I wasn't living up to my picture of what a good person is, and covered up the shame with anger"). But if an LLM were to say, "I'm…

> But if an LLM were to say, "I'm sad" or "I'm happy", does that actually correlate to anything? I'm OK with saying "The LLM was sad", if there is an internal state that leads to observable changes in behavior correlating with the kinds of changes in behavior humans have when they're sad. The question is, if the LLM says "I'm sad", is that because it has that internal state (self-reflection)? Or is it because that's the kind of thing a human would say in that context?

If they are not trained specifically to find those internal state correlates when introspecting, I would find it quite shocking to see that introspection is an emergent behavior of LLMs

Re: LLMs and self-referentiality

#89

Post by heavenlyblue »

thatjoeoverthr wrote: post #61
roenxi wrote: post #48

Earlier quoted context omitted.

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

It’s really not like that. If I ask you to tell me about a geographical place I just made up, you can trivially and generally instantly recognize you don’t recognize it. A child can do this. You wouldn’t be able to hold a job or generally get through life without this level of self awareness.

I had a great game as a kid where me and girls were playing for kisses on "words", i.e. "start the next word with the letter of the word I just said". So I invented a whole new vocabulary and then taught them that vocabulary. Apart from the fact that I got a lot of kisses from them they were so happy they learned something new they immediately went to share this with their (and my) parents. Obviously I got a slap but I just don't get how you can think "a kid can do this". You're either completely ignorant of different cultures (i.e. as an Eqstern European I still find it hard to remember Indian names, let alone tell if they are actually fucking with me) or intentionally simplifying the problem.
Post reply on HN