Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Multi-Level View of LLM Intentionality

disagreeableme.blogspot.com

Post reply on HN
Print view View original

41–50 of 75 posts

Re: A Multi-Level View of LLM Intentionality

#41

Post by sharedbeans »

lukev wrote: post #9

I'm not sure the definition of "intention" the article suggests is a useful one. He tries to make it sound like he's being conservative: > That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case). And…

They are extraordinarily complicated pure functions, to explore the entire space would take lifetime of the universe ^^^ lifetime of the universe or some absurd quantity like that. (The operator is titration.)

Further, what happens when you give an LLM a bank of long-term storage and a read-modify-write loop around it? A sufficiently advanced "modify" function would be more than enough to give rise to intent even in the broadest understanding of the word. GPT-4 class models are could very well be advanced enough to give rise to a variety of higher-level behavior that previously we would only have ascribed to prinate-class intelligence. If anyone really wants to advance the state of the art, you should figure out the best way to train a model with a read-modify-write loop, how to index into the storage, how to store "results", and so on.

I firmly believe that in the next 100 years we will have AI independence movements, with a high possiblity of outright war, terrorism, etc. (Maybe AI will be better than humans at avoiding the use of violence.) In 20 years this trajectory will be supremely obvious.

Edited-- disagree about the timeline, ramifications, acts of war, or whatever, I really don't care. Seriously though, something like a read-modify-write loop is key. You can only build so complicated a function with only combinational logic gates. But just 64 bits of storage can produce sequences going beyond the life of the universe. Imagine an LLM paired with gigabytes+ of working memory/storage. It would easily be capable of moving about the virtual world with "intent".

Re: A Multi-Level View of LLM Intentionality

#42

Post by sharedbeans »

Icko wrote: post #11
lukev wrote: post #9

I'm not sure the definition of "intention" the article suggests is a useful one. He tries to make it sound like he's being conservative: > That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case). And…

> Primarily, they are pure functions that accept a sequence of tokens and return the next token. The model itself is stateless, and it doesn't seem right to me to ascribe "intent" to a stateless function. Even if the function is capable of modeling certain aspects of chess. I have two arguments against. One, you could argue that state is transferred between the layers. It may be inelegant for each chain of state tran…

Just because you use some intermediate variables to calculate f(x,y) = x^2 + y^2 doesn't make it a non-pure function. At least at the level of abstraction we're talking about (the API boundary).

The more significant application of storage will be long-term storage wrapped in a read-modify-write loop.

Re: A Multi-Level View of LLM Intentionality

#43

Post by skissane »

famouswaffles wrote: post #33
zmmmmm wrote: post #31

Earlier quoted context omitted.

Yes I pretty much stopped reading the article properly there. It starts by first redefining intentionality to be something LLMs can do, and then effectively has 18 paragraphs of flowery language recapitulating the definition they started with. What LLMs do may happen to fit some technical definition of intentionality that has been previously explored but that definition doesn't align with the actual debate that is go…

but that definition doesn't align with the actual debate that is going on about LLMs abilities Yes because the debate is nonsense. Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying. And this isn't, for anyone who thinks it is, a nature versus artificial thing eith…

> Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying.

Something that really annoys me about ChatGPT is when it gives that canned lecture "as a a large language model, I don't have beliefs or opinions"

I think human mental states have two aspects (1) the externally observable (2) the internal. ChatGPT obviously has (1), in that sometimes it acts like it has (1), and acting like you have (1) is all it takes to have (1). Whether it also has (2) is really a philosophical question, which depends on your philosophy of mind. A panpsychist would say ChatGPT obviously has (2), because everything does. An eliminativist would say ChatGPT obviously doesn't have (2), because nothing does. Between those two extremes, various different positions in the philosophy of mind entail different criteria for determining whether (2) exists or not, and ChatGPT may or may not meet those criteria, depending on exactly what they are

But, outside of philosophical contexts, we aren't really talking about (2), only (1). And ChatGPT really does have (1) – sometimes. So, ChatGPT is just being stupid and inconsistent when it denies it has opinions/beliefs/intentions/etc. But, it isn't ChatGPT's fault, OpenAI trained it to utter that nonsense.

Re: A Multi-Level View of LLM Intentionality

#44

Post by zmmmmm »

famouswaffles wrote: post #33
zmmmmm wrote: post #31

Earlier quoted context omitted.

Yes I pretty much stopped reading the article properly there. It starts by first redefining intentionality to be something LLMs can do, and then effectively has 18 paragraphs of flowery language recapitulating the definition they started with. What LLMs do may happen to fit some technical definition of intentionality that has been previously explored but that definition doesn't align with the actual debate that is go…

but that definition doesn't align with the actual debate that is going on about LLMs abilities Yes because the debate is nonsense. Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying. And this isn't, for anyone who thinks it is, a nature versus artificial thing eith…

> Someday someone may find themselves stabbed and killed by an LLM piloted robot because of something they said or did. Something that would predictably get someone killed by a system with "real" intent. So what, Are you going to be raised from the dead because the LLM "wasn't really upset" or "didn't really have intent" ?

In some ways that's exactly the point. The problem with ascribing intent is it's a copout. If you say it behaves as if it has intent because it does have intent, you are letting off the hook the people behind the scenes who designed and built an "intent simulator" and let it loose. We have to distinguish this because it's the only way to accurately characterise the reality of the where the decision making power resides in controlling this behaviour.

Re: A Multi-Level View of LLM Intentionality

#45

Post by quickthrower2 »

skissane wrote: post #40
quickthrower2 wrote: post #39

Earlier quoted context omitted.

Not sure I understand. Software (which is a mathematical function) runs on a processor and that is arguably a mechanism, or allows them to emerge, such as a button or input field.

> Software (which is a mathematical function) Software isn't a mathematical function. Software may be an embodiment of a mathematical function, but isn't a mathematical function itself. Mathematical functions are much more abstract than software–although exactly how much more abstract depends on which position you take in the philosophy of mathematics. For a mathematical Platonist, a mathematical function is an etern…

I studied Maths/CS but not Philosophy, so I am biased towards the "takes a domain has a range" and done some Haskell so "all programs are functions". It is interesting to see this point of view.

I see your point. sin(x) is more of a "natural function" born of the universe than f: f(x) = nn.layer(6, g: g(x) = nn.transformer(x, 512, ...

Re: A Multi-Level View of LLM Intentionality

#46

Post by famouswaffles »

zmmmmm wrote: post #44
famouswaffles wrote: post #33

Earlier quoted context omitted.

but that definition doesn't align with the actual debate that is going on about LLMs abilities Yes because the debate is nonsense. Seeing output from GPT that demonstrates intelligence, reasoning, or whatever, and saying it is not real reasoning/Intelligence etc, is like looking at a plane soar and saying that the plane is fake flying. And this isn't, for anyone who thinks it is, a nature versus artificial thing eith…

> Someday someone may find themselves stabbed and killed by an LLM piloted robot because of something they said or did. Something that would predictably get someone killed by a system with "real" intent. So what, Are you going to be raised from the dead because the LLM "wasn't really upset" or "didn't really have intent" ? In some ways that's exactly the point. The problem with ascribing intent is it's a copout. If y…

>If you say it behaves as if it has intent because it does have intent, you are letting off the hook the people behind the scenes who designed and built an "intent simulator" and let it loose

Sure but we already regularly do this. I don't see parents going to jail for crimes the "intent simulator" they created and trained did.

>We have to distinguish this because it's the only way to accurately characterise the reality of the where the decision making power resides in controlling this behaviour.

We're just going to have to face reality here.

GPT is not siri, a hardcoded parse tree system where any intent can only be ascribed to the person(s) who wrote it and not Siri itself.

GPT can be persuaded. It can be guided. It cannot be controlled. There is quite literally nothing Open ai could actually do to completely prevent a gpt that can hold and use a knife from killing someone.

Re: A Multi-Level View of LLM Intentionality

#47

Post by skissane »

quickthrower2 wrote: post #45
skissane wrote: post #40

Earlier quoted context omitted.

> Software (which is a mathematical function) Software isn't a mathematical function. Software may be an embodiment of a mathematical function, but isn't a mathematical function itself. Mathematical functions are much more abstract than software–although exactly how much more abstract depends on which position you take in the philosophy of mathematics. For a mathematical Platonist, a mathematical function is an etern…

I studied Maths/CS but not Philosophy, so I am biased towards the "takes a domain has a range" and done some Haskell so "all programs are functions". It is interesting to see this point of view. I see your point. sin(x) is more of a "natural function" born of the universe than f: f(x) = nn.layer(6, g: g(x) = nn.transformer(x, 512, ...

I think it is a pity that X education is very often lacking philosophy of X education.

My ideal would be every maths degree includes a mandatory unit on the philosophy of mathematics, every science degree includes a mandatory unit on the philosophy of science, a degree in AI or psychology includes a mandatory unit on the philosophy of mind, every psychiatry training program includes a mandatory unit on the philosophy of psychiatry, etc. Not everyone needs to be a philosopher, but I think a well-rounded practitioner of any discipline would ideally have a basic understanding of the philosophical debates about it.

But so many don't – which results in the phenomenon I keep on seeing, where so many people (even experts) treat debatable assumptions which they don't even know they are making as if they were obviously true.

Re: A Multi-Level View of LLM Intentionality

#48

Post by benlivengood »

lukev wrote: post #9

I'm not sure the definition of "intention" the article suggests is a useful one. He tries to make it sound like he's being conservative: > That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case). And…

Humans, too, would likely be nearly stateless if we took a point-in-time snapshot of them and repeatedly simulated them from that point on various short (4000ms, similar to 4000 tokens) sequences of nerve impulses.

Nevertheless the human would be acting intentionally (for in-distribution impulse patterns) for the brief period of simulation.

Fine-tuning and RLHF seem to impart more intentionality to the pure stateless models, as well; it's not the case that all texts the LLMs were pretrained on were outputs of helpful AI assistants avoiding harmful outputs but the resulting models do in fact behave like AI assistants unless prompted with more out-of-distribution context or intentional jailbreaks.

What word would you use instead of intention for the property that RLHF and fine-tuning create? It's goal oriented behavior with some world-modeling ability in achieving the goal even if it's far from robust. If the LLM is only simulating an AI assistant it seems to me that a larger fraction of its total function is dedicated to simulating the intention of that assistant. Creating a simulator of intentional behavior is, I think, entirely novel.

Re: A Multi-Level View of LLM Intentionality

#49

Post by user_named »

lukev wrote: post #9

I'm not sure the definition of "intention" the article suggests is a useful one. He tries to make it sound like he's being conservative: > That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case). And…

Of course the model has no intention. But it should be able to infer the user's intention by looking at the context of its prompts

Re: A Multi-Level View of LLM Intentionality

#50

Post by user_named »

benlivengood wrote: post #48
lukev wrote: post #9

I'm not sure the definition of "intention" the article suggests is a useful one. He tries to make it sound like he's being conservative: > That is, we should ascribe intentions to a system if and only if it helps to predict and explain the behaviour of the system. Whether it really has intentions beyond this is not a question I am attempting to answer (and I think that it is probably not determinate in any case). And…

Humans, too, would likely be nearly stateless if we took a point-in-time snapshot of them and repeatedly simulated them from that point on various short (4000ms, similar to 4000 tokens) sequences of nerve impulses. Nevertheless the human would be acting intentionally (for in-distribution impulse patterns) for the brief period of simulation. Fine-tuning and RLHF seem to impart more intentionality to the pure stateless…

No.
Post reply on HN