From the OP's blog, his thoughts on AI safety:
Eight things to know about large language models [pdf]
21–30 of 114 posts
Re: Eight things to know about large language models [pdf]
#22This is a personal correspondence typeset via LaTeX — it is not an academic paper, and it was not peer-reviewed. (The document does not claim otherwise, but I think it's common for people to assume that documents that have been typeset in such a format are more rigorous than this is.) Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins: > There is increasingly subs…
LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382, which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this board state is used, because if you patch activation so that board state is modified, the model generates moves for this modified board state.
Re: Eight things to know about large language models [pdf]
#23Earlier quoted context omitted.
What does “reason” mean? It seems like it does everything I expect from something that reasons.
I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results…
This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
Re: Eight things to know about large language models [pdf]
#24Earlier quoted context omitted.
People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are sta…
New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.
Re: Eight things to know about large language models [pdf]
#25Earlier quoted context omitted.
> Gaining capabilities unpredictably is a bit exaggerated - it's more that a lot of apparent capabilities are embedded in language and these models approximate the truly vast amounts of text they've digested (imo). The fact is that they acquired abilities no one expected them to acquire. In hindsight you can say it was embedded in language and maybe you could have seen it coming, but it is an empirical fact that this…
It's still fairly exaggerated, because the LLM's purpose is basically "knowing human communication" and it still only exhibits that, but it knows how humans would respond if they obediently followed directions given to it exactly as instructed. And it was only unexpected to those that weren't following the news the GPT 3 paper was published in 2020 and detailed this wild advancement in its capabilities[0,1]. 0: https…
> but it knows how humans would respond if they obediently followed directions given to it exactly as instructed.
this framing denies agency, denies diversity, denies choice. A human, presented with the sort of directions and instructions that LLM's receive, does not have a single "correct" outcome (for that matter, neither do some LLMs). So no, an LLM does not "know how humans would respond". Instead, it can generate something that lies somewhere within the window of possibilities that a human might offer under similar conditions.
Re: Eight things to know about large language models [pdf]
#26This is a personal correspondence typeset via LaTeX — it is not an academic paper, and it was not peer-reviewed. (The document does not claim otherwise, but I think it's common for people to assume that documents that have been typeset in such a format are more rigorous than this is.) Leaving that aside, I really take issue with the style used by the author. For example, section 3 begins: > There is increasingly subs…
LLMs clearly develop internal representations, this is an empirical fact. For example, one paper cited is https://arxiv.org/abs/2210.13382 , which trained an LLM on Othello games, where an 8 layer model is fed one of 60 tokens (there are 60 possible moves in Othello, 8*8 minus 4 initially occupied squares). You can probe this model at layer 5, and it computes Othello board state from moves, and you can be sure this b…
> LLMs clearly develop internal representations, this is an empirical fact.
you've offered an anecdote. Some LLMs (more generally, this type of neural network model) will generate configurations that can be understood as a representation; others will not. The fact that the authors were able to find an apparent model in a heavily rule-based system is not incredibly surprising but offers little clue about whether this is "clearly" true for similar architectures that represent very different (real) world systems.
From your citation itself:
> More broadly, it would be interesting to study how our results generalize to models trained on natural language. One stepping stone might be to look at language models whose training data has included game transcripts. Will we see similar representation of board state?
Re: Eight things to know about large language models [pdf]
#27Earlier quoted context omitted.
What does “reason” mean? It seems like it does everything I expect from something that reasons.
I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results…
Does it really provide the statistically most common next token? I don’t think that’s even true. And we’ve seen what looks like emergent behavior as parameters increase. I suspect it is actually creating abstractions at a deeper level than standard DNNs.
Re: Eight things to know about large language models [pdf]
#28Earlier quoted context omitted.
> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are sta…
New bar for people claiming LLMs can't reason: invent a specific, testable problem, representable in text, that many humans can solve and LLMs can't, and tell us what it is.
… and then perform a careful search of books and the whole internet to be sure what you think is novel hasn’t been thoroughly debated somewhere on stackexchange.
Re: Eight things to know about large language models [pdf]
#29Earlier quoted context omitted.
I’ve been imagining “reasoning” as meaning “referring to an abstraction”. The idea being if you wanted to write a script to tell you how far an arrow of a given weight will fly, you’ll probably use some physics simulation (or even just a coefficient to multiply the input weight by) to abstract the arrow in flight. Then you can answer questions around that abstraction. The alternative is having 1000 different results…
This is one of those comments that begs for actually using LLMs. Yes, an advanced LLM like GPT-4 will absolutely show evidence of an abstracted world model, correctly answering questions about where person X thinks object Y is, or what relation two people have to each other, or playing a text adventure game while clearly following an internal map it has created for itself.
So I get that the evidence is that you can do insanely complex things with consistency, like playing a text adventure. You pick up a thing and it stays in your inventory etc, ergo, it’s modelled a character with an inventory in a text adventure.
vvv
But, coming at this as *respectfully and curiously* as I can here:
^^^
I guess I’m a little more skeptical about attributing magic to something that I know is only working from words as a source.
Like if it’s seen NOUN really close to VERB a lot, it can “assume” (read: bias output) that NOUN will VERB in the context it sees it in, that’s an abstraction! But a weak one right? Language is pretty inaccurate. The fact that NLP was a whole booming research field a few years ago tells me that just parsing language consistently is a difficult problem.
So, I know I can’t give you any convincing argument there since we’re already mentioning how people are giving untestable definitions of “reasoning” and “abstraction” in this thread. And im no better in that respect here, BUT I guess my best inkling of what feels “off” is a lack of precision, inherent to the medium in which it operates which makes it feel like a true abstraction is impossible in an LLM.
Re: Eight things to know about large language models [pdf]
#30Earlier quoted context omitted.
People routinely make up their own vague and ill defined meanings of understanding and reasoning to disqualify LLMs. This is necessary because LLMs obviously reason and understand by any evaluation that can be carried out. Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested.
> Seriously just watch. He's not actually going to be able to coherently define his "reasoning" in a way that can be tested. Google gives the following definition of the verb "reason": > think, understand, and form judgments by a process of logic. LLMs do not think, they do not understand, and they do not form judgments. They do not come to their own conclusions. They do not have the physical capability. They are sta…
We're survival machines, nothing more...
Complex behavior arises from simple systems all the time. You can't prove that these systems don't reason, no matter how loudly thou doth protest.