Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs and self-referentiality

scottaaronson.blog

Post reply on HN
Print view View original

61–70 of 95 posts

Re: LLMs and self-referentiality

#61

Post by thatjoeoverthr »

roenxi wrote: post #48
thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

It’s really not like that. If I ask you to tell me about a geographical place I just made up, you can trivially and generally instantly recognize you don’t recognize it. A child can do this. You wouldn’t be able to hold a job or generally get through life without this level of self awareness.

Re: LLMs and self-referentiality

#62

Post by thatjoeoverthr »

fyredge wrote: post #53
thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

From your comment, I noticed a sort of pattern that is often seen when discussing LLMs. That they are these amazing things that can run so fast they trip themselves in their attempts at achieving a task. So we resort to refining the models, creating guardrails, orchestrating harness, so as to alleviate the 'hallucination' problem. In contrast to human intelligence, there is an underlying mechanism that propels intell…

“trip themselves in their attempts at achieving a task”

I see this a lot in Claude Code. I assume it has to do with the training structure.

Example is “fallbacks”. Claude constantly sprinkles “fallbacks” in the code, even when I ask it not to. That is, write multiple candidate implementations into the same code with some kind of switch.

This is a problem because you only need one, and it would seem to have inflated the code for no reason. (The madness accelerates with code volume, so you must push back.) anyway, few people would do it this way.

But I thought, what could be the benefit?

If you’re being conditioned to pass evals one-shot with code that will be discarded and never read, it’s a great strategy. If you have more than one way to solve it, you can just put both. The behavior would easily be reinforced, if trained that way.

But in any case, again, a certain nature and certain conditioning.

I think we’ll learn to accept it as AGI but also that no intelligence is fully divorced from context, limits and conditioning.

Re: LLMs and self-referentiality

#63

Post by m-hodges »

I've used concepts from GEB a lot in my thinking about how LLMs.

On self-referentiality and the MU Puzzle: https://matthodges.com/posts/2025-04-21-openai-o4-mini-high-...

On Gödelian limits of prompt-safe AI : https://matthodges.com/posts/2025-08-26-music-to-break-model...

On the blurry line between pattern matching and reasoning: https://matthodges.com/posts/2026-08-19-bongard-problems/

I'm glad that the author of this post ends with:

> What’s left? Consciousness

because when I've read (and re-read) GEB, I find the book to be much deeper than just a threshold for conversational intelligence.

Re: LLMs and self-referentiality

#64

Post by DonaldFisk »

fc417fc802 wrote: post #57
DonaldFisk wrote: post #49

Earlier quoted context omitted.

I just explained this. If someone looks at the contents of an image file without displaying it, they don't experience the colours either. So how can LLMs experience them?

Right. Because your eyes ... imbue the magical pixie dust ... no wait, is it your visual cortex that does it? Hmm ... Less facetiously, I'm unclear where you believe this magic "qualia" thing to be happening and what you believe the (meta?)physical mechanism to be.

Quite a lot has been published about neural correlates of consciousness, so I'll go with whatever the evidence suggests.

There are literally hundreds of attempted explanations of phenomenal consciousness, none of which is generally accepted. For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. At the very minimum, I think to be conscious of anything (other than nothing) requires the ability to sense the physical environment.

What's your position?

Re: LLMs and self-referentiality

#65

Post by DonaldFisk »

fc417fc802 wrote: post #60
DonaldFisk wrote: post #54

Earlier quoted context omitted.

People aren't just consciousness, they're conscious of things. Of course a deaf or blind person is conscious, even if they've been deaf or blind since birth. But they aren't conscious of hearing or seeing things, and if you ask them they'll tell you that. I'm not conscious of magnetic fields, or ultraviolet light, but other animals are, and no amount of study will overcome that deficit.

Okay, but there are multi-modal models. So do those experience qualia of sound and images? Do the text based ones experience the qualia of reading, or of finally comprehending a difficult concept after struggling with it? For that matter, what exactly is the difference between "seeing" that a car is red versus an LLM reading an array of pixel data? Your eyes "just" translate photons into analog signals after all.

I've already answered this. There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of, and if you don't believe me you can ask them. Or by looking at a speech spectrogram (which, with practice, are more easily readable).

Or you can ask yourself what ultraviolet looks like, or what it's like to sense a magnetic field. Lots of animals can do these, and would know what it's like, but people can't.

Re: LLMs and self-referentiality

#66

Post by m-hodges »

hax0ron3 wrote: post #8

Was Hofstadter ever arguing that intelligence requires self-referentiality? I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality. I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His disc…

A few quotes towards the end of GEB:

> I see no barrier imposed by Gödel’s Theorem to the implementation on computers (or their successors) of types of symbol manipulation that achieve roughly the same results as brains do.

And a distinction between intelligence and consciousness:

> It is entirely another question to try and duplicate in a program some particular human’s mind—but to produce an intelligent program at all is a more limited goal.

And then:

> Gödel’s Theorem doesn’t ban our reproducing our own level of intelligence via programs

That is pretty hard to square with "GEB argued that AI couldn’t achieve intelligence without first mastering Gödelian self-reference."

A bit more introspective:

> I think that the process of coming to understand Gödel’s proof, with its construction involving arbitrary codes, complex isomorphisms, high and low levels of interpretation, and the capacity for self-mirroring, may inject some rich undercurrents and flavors into one’s set of images about symbols and symbol processing, which may deepen one’s intuition for the relationship between mental structures on different levels.

Re: LLMs and self-referentiality

#67

Post by roenxi »

thatjoeoverthr wrote: post #61
roenxi wrote: post #48

Earlier quoted context omitted.

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

It’s really not like that. If I ask you to tell me about a geographical place I just made up, you can trivially and generally instantly recognize you don’t recognize it. A child can do this. You wouldn’t be able to hold a job or generally get through life without this level of self awareness.

So can LLMs. I asked one about South Wollopop and it suggested that it'd never heard of it but maybe I meant South Wollo in Ethiopia. And that's just a local model with modest hardware and no internet access. First attempt.

I mean, if our position is that a hyper advanced statistical model is going to ultimately struggle with the concept of something being unlikely to be true then the statisticians may as well give up in despair. There is no theoretical obstacle here.

Re: LLMs and self-referentiality

#68

Post by fc417fc802 »

DonaldFisk wrote: post #64
fc417fc802 wrote: post #57

Earlier quoted context omitted.

Right. Because your eyes ... imbue the magical pixie dust ... no wait, is it your visual cortex that does it? Hmm ... Less facetiously, I'm unclear where you believe this magic "qualia" thing to be happening and what you believe the (meta?)physical mechanism to be.

Quite a lot has been published about neural correlates of consciousness, so I'll go with whatever the evidence suggests. There are literally hundreds of attempted explanations of phenomenal consciousness, none of which is generally accepted. For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. At the very minimum, I think to be conscious of anything (other than nothing) requires the…

> For what it's worth, I think it's fundamental, i.e. not an emergent property of matter.

> requires the ability to sense the physical environment

Aren't those two statements at odds? Your synapses are made of matter, so sensing the physical environment is being done by matter. You are fundamentally matter (AFAIK). To suggest otherwise the only thing that comes to mind is substance dualism.

My position is that we don't know. I know I'm conscious. I assume other humans are simply because they're approximately the same as me and were produced by the same process. The farther you get from humans the less certain I am. Many insects resemble biological automatons if you examine them closely. Certainly single celled organisms don't seem likely to have anything going on.

If you figure that a clod of dirt can't possibly be conscious but a mouse is, what's the relevant physical difference? Electricity? Computers have that. Complexity? Information density? Both of those as well.

What do you think it means to sense?

Re: LLMs and self-referentiality

#69

Post by fc417fc802 »

DonaldFisk wrote: post #65
fc417fc802 wrote: post #60

Earlier quoted context omitted.

Okay, but there are multi-modal models. So do those experience qualia of sound and images? Do the text based ones experience the qualia of reading, or of finally comprehending a difficult concept after struggling with it? For that matter, what exactly is the difference between "seeing" that a car is red versus an LLM reading an array of pixel data? Your eyes "just" translate photons into analog signals after all.

I've already answered this. There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of, and if you don't believe me you can ask them. Or by looking at a speech spectrogram (which, with practice, are more easily readable). Or you can ask yourself what ultraviolet looks like, or what it's like to sense a magnetic field. Lots of…

No, you have not answered. I asked you what you believe the difference is between the two phenomena in concrete terms. What you've done is beg the question.

> There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of

On what basis could you possibly make this claim? What does it mean to hear? You're taking in information and decoding it, right? So what magical pixie dust could your eardrum possibly be applying that specially privileges that particular mode of input?

How is an LLM "reading" pixel data fundamentally different from your eyes transforming it into an analog signal?

Please be specific about the mechanism.

Re: LLMs and self-referentiality

#70

Post by yeellow »

_hao wrote: post #55
yeellow wrote: post #37

Earlier quoted context omitted.

Do you treat LLM as algorithmic? I mean, the process of computing tokens is algorithmic, also the process of training, but does it mean that LLM itself follows an algorithm, in a practical sense? If yes, how is it different to a biological system constrained by physics?

A short disclaimer before answering. Right now I'm doing a Mathematics & Physics undergrad degree and have interest in philosophy. I'm not a specialist in any of those fields. I have plans to tackle the actual math around LLM's a bit further, but I'm not there yet. To question 1 based on my current understanding (and let's say "beliefs") is that - yes, LLM's are following a probabilistic algorithm to give me the outp…

LLM algorithm does not need to be probabilistic, we add probability to make it more interesting, but in principle it should return the same tokens for the same input (which is often desired, and we had temperature=0 for that, even if for practical reasons it was not always working that way). But it is not the point. My point is that LLM inference follows an easy algorithm, but what we get is something different, because weights are part of the algorithm and they are not easily interpretable. So if I ask LLM to write me an essay on a given subject I know what it is technically doing, but writing it as an algorithm different than "convert this text to tokens and than perform billions of simple operations on them to get the next token and repeat" seems hopeless. Because the black box of weights is what matters. In this meaning it is not an algorithm we can find in a book on algorithms. Of course it's a computational process, but the question is, if it is qualitatively different to the one in our heads.
Post reply on HN