Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

LLMs and self-referentiality

scottaaronson.blog

Post reply on HN
Print view View original

71–80 of 95 posts

Re: LLMs and self-referentiality

#71

Post by DonaldFisk »

fc417fc802 wrote: post #68
DonaldFisk wrote: post #64

Earlier quoted context omitted.

Quite a lot has been published about neural correlates of consciousness, so I'll go with whatever the evidence suggests. There are literally hundreds of attempted explanations of phenomenal consciousness, none of which is generally accepted. For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. At the very minimum, I think to be conscious of anything (other than nothing) requires the…

> For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. > requires the ability to sense the physical environment Aren't those two statements at odds? Your synapses are made of matter, so sensing the physical environment is being done by matter. You are fundamentally matter (AFAIK). To suggest otherwise the only thing that comes to mind is substance dualism. My position is that we don…

> Aren't those two statements at odds?

Sensing (i.e. picking up information about its environment through sense organs) is a physical phenomenon and is reasonably well-understood. Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it. Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism.

So what's conscious? People and other animals with sense organs and nervous systems. It's even possible that anything which senses its environment could be conscious: https://organism.earth/library/document/what-is-it-like-to-b... . That might include a robot connected to a camera and other sensory apparatus. Now you could argue that a computer running an LLM can pick up what its user types, and might therefore be conscious of which keys are being pressed. But not of the colours, sounds, smells, and other things described by human authors in its training set.

Re: LLMs and self-referentiality

#72

Post by jstanley »

DonaldFisk wrote: post #65
fc417fc802 wrote: post #60

Earlier quoted context omitted.

Okay, but there are multi-modal models. So do those experience qualia of sound and images? Do the text based ones experience the qualia of reading, or of finally comprehending a difficult concept after struggling with it? For that matter, what exactly is the difference between "seeing" that a car is red versus an LLM reading an array of pixel data? Your eyes "just" translate photons into analog signals after all.

I've already answered this. There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of, and if you don't believe me you can ask them. Or by looking at a speech spectrogram (which, with practice, are more easily readable). Or you can ask yourself what ultraviolet looks like, or what it's like to sense a magnetic field. Lots of…

But you can't experience the qualia of tokens coming in and LLMs can.

Re: LLMs and self-referentiality

#73

Post by _hao »

iainmerrick wrote: post #43
_hao wrote: post #25

Earlier quoted context omitted.

Scott Aaronson mentions Penrose's "The Emperor's New Mind", but I feel "Shadows of the Mind" is putting forward a much clearer view of Penrose's thesis. At the current stage of my life I'm quite comfortably in Camp C. "Intelligence" and "consciousness" are not algorithmic.[1] A lot of materialists are in Camp A. For some even today, LLM's are AGI. Unfortunately, the terminology is quite clearly not adequate. There's…

Do you distinguish between Penrose's argument and the older one by Lucas, invoking Gödel's proof that certain facts are non-computable? I always felt it's basically the same argument, and Lucas's was never really convincing; Hofstadter rebutted it very convincingly in GEB long before Penrose's books came out. To me, camp C's claim is possible but I don't buy the argument that says it's necessary. So I guess I'm camp…

I haven't read GEB (yet), but I'd say I do feel there's a difference between the original formulation of Lucas and the extensions that Penrose provide. At this stage I can't say whether I'd agree with Hofstadter or not on the rebuttal though.

As for the camps, I agree with you that B doesn't really make sense (at least to me). I can put myself in position to argue for A, C and D, but as mentioned in my original post so far I agree with C the most.

Re: LLMs and self-referentiality

#74

Post by DonaldFisk »

fc417fc802 wrote: post #69
DonaldFisk wrote: post #65

Earlier quoted context omitted.

I've already answered this. There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of, and if you don't believe me you can ask them. Or by looking at a speech spectrogram (which, with practice, are more easily readable). Or you can ask yourself what ultraviolet looks like, or what it's like to sense a magnetic field. Lots of…

No, you have not answered. I asked you what you believe the difference is between the two phenomena in concrete terms. What you've done is beg the question. > There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of On what basis could you possibly make this claim? What does it mean to hear? You're taking in information and…

> On what basis could you possibly make this claim?

People with absent qualia have been asked, e.g. Knut Nordby: https://www.consc.net/misc/achromat.html

"Although I have acquired a thorough theoretical knowledge of the physics of colours and the physiology of the colour receptor mechanisms, nothing of this can help me to understand the true nature of colours. From the history of art I have also learned about the meanings often attributed to colours and how colours have been used at different times, but this too does not give me an understanding of the essential character or quality of colours."

Knut Nordby was in the same position as an LLM as far as colours are concerned. He learned all he could but words didn't give him any idea of *what it's like to see them". LLMs have access to everything that's written about colours, but that didn't help Nordby.

Nordby's no longer around to ask about image files, but how do you think reading thems would help? He might then have been able to say, e.g. "This pixel is orange," but he still wouldn't have known what it's like to see orange. The same applies to a multimodal LLM.

Re: LLMs and self-referentiality

#75

Post by fc417fc802 »

DonaldFisk wrote: post #74
fc417fc802 wrote: post #69

Earlier quoted context omitted.

No, you have not answered. I asked you what you believe the difference is between the two phenomena in concrete terms. What you've done is beg the question. > There's no way someone deaf from birth could know what it's like to hear sounds, by looking at the pixels in a sound recording they know the format of On what basis could you possibly make this claim? What does it mean to hear? You're taking in information and…

> On what basis could you possibly make this claim? People with absent qualia have been asked, e.g. Knut Nordby: https://www.consc.net/misc/achromat.html "Although I have acquired a thorough theoretical knowledge of the physics of colours and the physiology of the colour receptor mechanisms, nothing of this can help me to understand the true nature of colours. From the history of art I have also learned about the mea…

I'm not claiming that any particular disabled person has managed to achieve it. I'm asking on what basis you claim it to be categorically impossible.

You've claimed that black swans are impossible, I've asked how you arrived at that conclusion, and your answer is that you looked and don't see any.

You're here claiming that what submarines do is somehow fundamentally different than swimming and that we know this because they don't have legs. You need legs to swim. And I'm questioning the soundness of your reasoning and you're pointing to cripples without legs who can't swim as evidence. It doesn't make any sense.

> He might then have been able to say, e.g. "This pixel is orange," but he still wouldn't have known what it's like to see orange. The same applies to a multimodal LLM.

How is a multimodal LLM taking in RGB data fundamentally different than what your eyes are doing? That is the core question that you keep dodging.

Re: LLMs and self-referentiality

#76

Post by fc417fc802 »

DonaldFisk wrote: post #71
fc417fc802 wrote: post #68

Earlier quoted context omitted.

> For what it's worth, I think it's fundamental, i.e. not an emergent property of matter. > requires the ability to sense the physical environment Aren't those two statements at odds? Your synapses are made of matter, so sensing the physical environment is being done by matter. You are fundamentally matter (AFAIK). To suggest otherwise the only thing that comes to mind is substance dualism. My position is that we don…

> Aren't those two statements at odds? Sensing (i.e. picking up information about its environment through sense organs) is a physical phenomenon and is reasonably well-understood. Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it. Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism. So what's conscious? Peo…

> Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it.

Aren't you inventing that whole cloth? What evidence is there to support this claim? Can you empirically measure consciousness to demonstrate its presence or absence?

Is there a lobotomy you can perform to make someone unconscious but still able to respond to those around them? If such a lobotomy is within the realm of physical possibility then (at least theoretically) why can't LLMs be given equivalent functionality? If such a lobotomy is fundamentally impossible then by what mechanism is consciousness influencing human behavior?

> Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism.

So on the one hand you claim it's an extra (ie independent) step but on the other earlier you claimed that it requires the ability to sense the physical environment and that somehow LLMs can't do this (despite that I can feed them data from an environmental sensor). If it's an extra step presumably arbitrary data can be fed into it. You're simultaneously claiming that it's separate, yet somehow fundamentally tied to sensing the physical world (so not actually separate), and also that certain types of physical sensors are specially privileged (no physical mechanism proposed). And you don't seem to have evidence supporting these seemingly contradictory claims.

Re: LLMs and self-referentiality

#77

Post by DonaldFisk »

fc417fc802 wrote: post #76
DonaldFisk wrote: post #71

Earlier quoted context omitted.

> Aren't those two statements at odds? Sensing (i.e. picking up information about its environment through sense organs) is a physical phenomenon and is reasonably well-understood. Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it. Consciousness is an extra step, and if it's fundamental then that is a kind of substance dualism. So what's conscious? Peo…

> Consciousness is non=physical and takes that information (after processing in the brain) and directly experiences it. Aren't you inventing that whole cloth? What evidence is there to support this claim? Can you empirically measure consciousness to demonstrate its presence or absence? Is there a lobotomy you can perform to make someone unconscious but still able to respond to those around them? If such a lobotomy is…

> Aren't you inventing that whole cloth?

I should have said, "I think that consciousness ..."

> So on the one hand you claim it's an extra (ie independent) step ...

I should have said, "The conscious experience (of qualia) is an extra step, and if consciousness is fundamental then that's a kind of substance dualism.

Thanks for pointing these out - I should have proof-read my reply before posting.

Re: LLMs and self-referentiality

#78

Post by CamperBob2 »

thatjoeoverthr wrote: post #61
roenxi wrote: post #48

Earlier quoted context omitted.

They symptoms sound a lot like humans, so I don't see how it stems from their lack of self reference. Most people you need to keep them in areas they understand or they go to pieces. The lack of self reference just means every time the context clears they reset. They are systems in a permanent state of extreme amnesia.

It’s really not like that. If I ask you to tell me about a geographical place I just made up, you can trivially and generally instantly recognize you don’t recognize it. A child can do this. You wouldn’t be able to hold a job or generally get through life without this level of self awareness.

Current frontier models are pretty good at telling you when they don't know something, or if you're asking about something nonexistent or nonsensical. Not always, but there has been massive improvement on this front lately.

If your last experience with frontier LLMs (read: not models that Google and ChatGPT are giving away for free, but models you have to pay for) was over a year ago, you may not realize that.

Re: LLMs and self-referentiality

#79

Post by DonaldFisk »

fc417fc802 wrote: post #75
DonaldFisk wrote: post #74

Earlier quoted context omitted.

> On what basis could you possibly make this claim? People with absent qualia have been asked, e.g. Knut Nordby: https://www.consc.net/misc/achromat.html "Although I have acquired a thorough theoretical knowledge of the physics of colours and the physiology of the colour receptor mechanisms, nothing of this can help me to understand the true nature of colours. From the history of art I have also learned about the mea…

I'm not claiming that any particular disabled person has managed to achieve it. I'm asking on what basis you claim it to be categorically impossible. You've claimed that black swans are impossible, I've asked how you arrived at that conclusion, and your answer is that you looked and don't see any. You're here claiming that what submarines do is somehow fundamentally different than swimming and that we know this becau…

> I'm asking on what basis you claim it to be categorically impossible.

Because the neural correlates of consciousness wouldn't exist in their brain. I've even looked into what happens if someone, blind from birth, takes LSD. There aren't any large scale studies as it's very difficult to get approval for them, but the short answer appears to be that they don't see things: https://www.livescience.com/62343-psychedelics-lsd-effects-b...

> How is a multimodal LLM taking in RGB data fundamentally different than what your eyes are doing?

The multimodal LLM is a program run as a process on a computer, which isn't aware of what it's running: it's just blindly, repeatedly, executing whatever instruction is at its program counter. The LLM itself isn't doing anything. It's just sitting somewhere in memory being executed by the computer. Brains, on the other hand, don't run programs. Each part of the brain has a special purpose, and somewhere in it there are neurons which correlate with what is being experienced whenever something is looked at.

Re: LLMs and self-referentiality

#80

Post by [deleted] »

monideas wrote: post #28

> It claimed (incorrectly, in my view and in most experts’) that AI could never work because there was something about Gödel’s Theorem and self-reference that no computer program could ever capture The fact he uses an appeal to “the experts” here for something that is fundamentally a philosophical or metaphysical claim shows he doesn’t actually understand the point. If LLMs were equivalent to humans then we wouldn’t…

[deleted]
Post reply on HN