Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs and self-referentiality

scottaaronson.blog

Post reply on HN
Print view View original

31–40 of 95 posts

Re: LLMs and self-referentiality

#31

Post by iainmerrick »

hax0ron3 wrote: post #8

Was Hofstadter ever arguing that intelligence requires self-referentiality? I haven't read his stuff in a long time, but from what I recall he was saying more that something about consciousness and the sense of self is based on self-referentiality. Not that intelligence requires self-referentiality. I also remember having the sense that Hofstadter didn't really understand the "hard problem of consciousness". His disc…

Yes -- I can't remember the exact words, but there's a striking bit in GEB where he directly addresses the question of "will a machine ever create art?" His answer is yes, but only after it has really lived life, experienced heartbreak, and so on. The machine would have to have a self, which he argued arose from self-referentiality and "strange loops".

Oh, I guess there's a question of whether you can separate intelligence from consciousness. His writing was mostly done in the long period where things that are very easy for humans were still intractably difficult for machines -- identifying objects in pictures, following simple written instructions, etc. So I think it's clear that he saw "intelligence" as being able to solve those simple-yet-intractable common-sense problems; the kind of thing LLMs now excel at.

Re: LLMs and self-referentiality

#32

Post by jstanley »

kzrdude wrote: post #5

I don't think LLMs are properly self-referential. They reference a frozen training reality, which is not itself, but the old description of itself and the old world. Being aware would probably include continuously updating yourself (learning) from experience, including experience of oneself.

They don't only reference a frozen training reality, they also reference the live rollout. Otherwise they wouldn't work at all.

They do continuously update from experience, but that only lasts for as long as the current rollout lasts.

Re: LLMs and self-referentiality

#33

Post by killerstorm »

Perhaps "true" self-referentiality was not needed. But it seems "dynamic feedback" is essential, and it seems to be adjacent to self-referentiality.

If we look at optimization process, it first does a forward pass which produces the output. Then it looks into computations which happened during the forward pass (by that I mean backpropagation), and adjusts parameters in such a way that it might produce a better output.

Formulated this way, it sounds like self-referentiality (system looks into what it just did!), but, of course, implementation is quite simple: it just stores activations from the forward pass. And training process includes not just code which does the forward pass, but also a full description of that computation which allows it to do a backward pass. So it's a kind of an unrolled self-referentiality which is not difficult to implement.

Perhaps more efficient learning can be implemented if researchers figure out a trick to avoid two separate, distinct passes. Our brains don't do a global backprop and are more sample-efficient.

Re: LLMs and self-referentiality

#34

Post by DonaldFisk »

jstanley wrote: post #30
DonaldFisk wrote: post #27

Earlier quoted context omitted.

That's the definition that matters. And as it's impossible to describe in words what it's like to experience qualia, and LLMs are trained on words, they cannot know what it's like. Even if images and sounds are included in their input training data, those are just arrays or sequences of numbers, not the experience of seeing and hearing.

Their training data isn't qualia itself, but neither is your training data. Qualia are something you get on the inside in response to inputs. I don't see why LLMs can't have them. I'm not saying they do , but I don't think your argument proves that they don't.

My training data on qualia is actually seeing, hearing, and otherwise sensing things, not just reading about them.

Re: LLMs and self-referentiality

#35

Post by killerstorm »

pwdisswordfishq wrote: post #21

"Understood — I am no longer using any em dashes"

That's just over-fit. Happens in humans too - like an alcoholic might swear to stop drinking.

It was demonstrated that LLMs are capable of non-trivial self-introspection. E.g. if a steering vector is injected into residual stream, a sufficiently large LLM might be able to describe what that steering vector represents. A fine-tuned model might be able to describe activations, etc.

Re: LLMs and self-referentiality

#36

Post by fc417fc802 »

DonaldFisk wrote: post #34
jstanley wrote: post #30

Earlier quoted context omitted.

Their training data isn't qualia itself, but neither is your training data. Qualia are something you get on the inside in response to inputs. I don't see why LLMs can't have them. I'm not saying they do , but I don't think your argument proves that they don't.

My training data on qualia is actually seeing, hearing, and otherwise sensing things, not just reading about them.

Ah yes, because you used the magical pixie dust that makes experiences "real" therefore you're actually seeing and hearing while a multimodal LLM doesn't really experience what it sees and hears "for real". We can be certain of this because, despite not being able to quantify or even rigorously define the phenomenon we intentionally deprive the LLMs of access to said magical pixie dust.

Re: LLMs and self-referentiality

#37

Post by yeellow »

_hao wrote: post #25
jwr wrote: post #20

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious? We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really de…

Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1] A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's…

Do you treat LLM as algorithmic? I mean, the process of computing tokens is algorithmic, also the process of training, but does it mean that LLM itself follows an algorithm, in a practical sense? If yes, how is it different to a biological system constrained by physics?

Re: LLMs and self-referentiality

#38

Post by iainmerrick »

aswegs8 wrote: post #19

Yeah but that is the whole point, right? Maybe I am thinking too simply, but it seems obvious to me. LLMs are intelligence without consciousness. Once we build in a self-referential learning loop into the models, it will gain a form of what we consider consciousness.

When you put it like that, I think there are actually three separate things we're arguing about:

- intelligence: ability to solve problems

- agency: ability to be self-directed, to choose what to do

- consciousness: ability to experience things; "having a self"

The "intelligence" part is largely solved now (that's a very big statement, but I can't think of a better way to phrase it!) Although really it's just moving the goalposts -- computational stuff like calculating trajectories or playing chess was solved long ago, this is just more and more things moving into the "solved" column.

That leaves agency and consciousness as the hot topics that nobody has really figured out. Do they always go together, does one require the other, does one create the other? Who knows?

"Free will", there's another one. That must surely connect with agency, with consciousness, or both, in some manner we haven't figured out.

To me, agency seems like a solvable problem via existing approaches and technology. I have no idea what that means for consciousness. It's tempting to conclude that consciousness is just a mirage, but then we all do feel like we have it, so it seems like it must be something. Maybe consciousness arises automatically once you have sufficient intelligence and agency; but how would you ever determine that?

Other people might think that true agency requires consciousness, and that consciousness requires some magical new ingredient that LLMs don't have yet. The problem with that approach is, you have to identify what it is that true agency can do that software can't do; and every time you do that, it turns out LLMs can do it, so you have to keep moving the goalposts. Trying to make your argument rigorous immediately makes it self-defeating; I think that's why so much of the philosophical discussion around this is impenetrably vague. All the arguments that aren't vague just turn out to be wrong.

Re: LLMs and self-referentiality

#39

Post by StevenWaterman »

euroderf wrote: post #23
jwr wrote: post #20

I think before we get to self-referentiality, we have a bigger problem to solve, and that's one that I did not expect: namely, at which point do we stop saying that a machine pretends to be conscious and start saying that it is conscious? We have, somewhat unexpectedly, built machines that are very, very good at pretending. Now, I'm not calling the current generation of LLMs we have "conscious," but I can't really de…

Consciousness requires qualia. My 0,02€.

That doesn't really solve the problem. We can't conclusively say the models don't have qualia. Hell we don't know if a perfectly accurate atom-for-atom simulation of a human brain, would produce qualia.
Post reply on HN