Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs and self-referentiality

scottaaronson.blog

Post reply on HN
Print view View original

41–50 of 95 posts

Re: LLMs and self-referentiality

#41

Post by imtringued »

What the author refers to is the ability to iterate and update the internal memory. A classic transformer based LLM can only produce the next token and never go back and update old tokens or delete them. The best thing it can do is produce thinking tokens to serialize the internal state of the final layer so that it can pass it back into the first layer.

Just like the transformer was an advancement over LSTMs by making it possible to have perfect recall (reading every input), the only way to improve over the transformer is to build a deep equilibrium version, where the DEQ transformer is capable of updating its own memory (writing every output).

Such a machine would be considered a linear bounded automaton (a turing machine without unlimited tape) and therefore even the human brain could not have an architectural edge over it in terms of intelligence. The human brain could only have an edge in terms of energy efficiency.

Re: LLMs and self-referentiality

#42

Post by k__ »

thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

Any signal that allows the model to see what's wrong helps.

Checking code is (relatively) easy, you can use static type checks, linters, and execute it to see if it's correct.

Fact checking is harder. A RAG can only check what's in the database, so you have to know what to know beforehand.

Re: LLMs and self-referentiality

#43

Post by iainmerrick »

_hao wrote: post #25
jwr wrote: post #20

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious? We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really de…

Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1] A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's…

Do you distinguish between Penrose's argument and the older one by Lucas, invoking Gödel's proof that certain facts are non-computable?

I always felt it's basically the same argument, and Lucas's was never really convincing; Hofstadter rebutted it very convincingly in GEB long before Penrose's books came out.

To me, camp C's claim is possible but I don't buy the argument that says it's necessary. So I guess I'm camp A. (D is religion and B seems to me incoherent.)

Re: LLMs and self-referentiality

#44

Post by lanyard-textile »

jwr wrote: post #20

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious? We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really de…

Our definition of consciousness requires stakes.

We associate consciousness with precious life: Life that comes with rights, needs, wants.

Life has rights because it naturally demands to be continued. If it makes no difference whether it is continued, it needs no rights, and we do not consider it conscious.

Life has needs because it cannot survive without. If it has no needs, it has no death. No death, no life.

Life has wants because it is harder without. If it has no wants, it has no emotion: It cannot feel pain, otherwise it would want no pain. It cannot feel happiness, otherwise it would want more happiness.

We don't want to build consciousness! :) Deliberately designing pain and artificial needs is a cruel exercise. We can build that, but we see no need to.

The real question at the table is, are these minds? And can minds exist without consciousness? Unequivocally, yes.

Re: LLMs and self-referentiality

#45

Post by ccppurcell »

Weird side swipe at Marx. Or not so weird I guess. Correct me if I'm wrong but I doubt Marx ever predicted that a workers' uprising would lead to classless utopia. It's clearly a necessary but not sufficient condition.

Re: LLMs and self-referentiality

#46

Post by t_von_doom »

fc417fc802 wrote: post #36
DonaldFisk wrote: post #34

Earlier quoted context omitted.

My training data on qualia is actually seeing, hearing, and otherwise sensing things, not just reading about them.

Ah yes, because you used the magical pixie dust that makes experiences "real" therefore you're actually seeing and hearing while a multimodal LLM doesn't really experience what it sees and hears "for real". We can be certain of this because, despite not being able to quantify or even rigorously define the phenomenon we intentionally deprive the LLMs of access to said magical pixie dust.

I agree. To further ask, why is your training data objectively real and qualifies you for consciousness?

Is a deaf or blind person not conscious because they don't share your training data? If a deaf person reads about sounds, their experience of them is moot?

If an alien with a greater array of senses then us exists, are we therefore not conscious?

Re: LLMs and self-referentiality

#47

Post by t_von_doom »

yeellow wrote: post #37
_hao wrote: post #25

Earlier quoted context omitted.

Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1] A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's…

Do you treat LLM as algorithmic? I mean, the process of computing tokens is algorithmic, also the process of training, but does it mean that LLM itself follows an algorithm, in a practical sense? If yes, how is it different to a biological system constrained by physics?

I would be interested in the answer to this question too. The visual cortex is certainly algorithmic in that light data enters, various cells and synapses react and fire and that information is ingested into something 'conscious'. Where does this then exist, and would it even be able to exist if we removed the biological algorithms at play?

Re: LLMs and self-referentiality

#48

Post by roenxi »

thatjoeoverthr wrote: post #12

Their lack of self reference is a core problem that undergirds a lot of faults that do occur during inference, but their breadth + the agent harness successfully covers it well, so it requires a bit of poking to witness. The “hallucination” phenomenon is exactly this. They don’t know the scope of their own knowledge, and they just say stuff, so if you go out of band, it has a higher probability emitting claims that a…

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

Re: LLMs and self-referentiality

#49

Post by DonaldFisk »

fc417fc802 wrote: post #36
DonaldFisk wrote: post #34

Earlier quoted context omitted.

My training data on qualia is actually seeing, hearing, and otherwise sensing things, not just reading about them.

Ah yes, because you used the magical pixie dust that makes experiences "real" therefore you're actually seeing and hearing while a multimodal LLM doesn't really experience what it sees and hears "for real". We can be certain of this because, despite not being able to quantify or even rigorously define the phenomenon we intentionally deprive the LLMs of access to said magical pixie dust.

I just explained this. If someone looks at the contents of an image file without displaying it, they don't experience the colours either. So how can LLMs experience them?

Re: LLMs and self-referentiality

#50

Post by gwd »

Two comments on this, trying to take a "which hypothesis fits the evidence" approach.

First, an LLM describing its own experience is not actually proof that it has any experience to be aware of, any more than an LLM confidently asserting any other fact means that it knows that fact is true. LLMs will describe music or tastes, in spite of the fact that it's never actually heard or tasted anything, based only on what it's read about them. In the same way, "non-aware spicy autocomplete" would produce an LLM that spoke about its own experience, based only on the input it has of people speaking about their own experience.

That said / secondly, from the little I understand of LLM architecture, I believe there are a large number of self-referential mechanisms built in. For one, nearly all transformers have a "residual layer", with various neural networks essentially reading and modifying it. This effectively forms a loop. Additionally, the "thinking" mechanism allows it to read what it's written and generate more things, which is again a loop.

So, maybe people didn't think, "Hey, we should build some loops, maybe that will make it conscious". But if "strange loop" is what defines consciousness, there are lots of loops in there onto which such a strange loop could conceivably form.

Post reply on HN