From word models to world models
arxiv.org
From word models to world models
1–10 of 119 posts
Re: From word models to world models
#2After a quick/superficial read, my understanding is that the authors:
(a) induce an LLM to take natural language inputs and generate statements in a probabilistic programming language that formally models concepts, objects, actions, etc. in a symbolic world model, drawing from a large body of research on symbolic AI that goes back to pre-deep-learning days; and
(b) perform inference using the generated formal statements, i.e., compute probability distributions over the space of possible world states that are consistent with and conditioned on the natural-language input to the LLM.
If this approach works at a larger scale, it represents a possible solution for grounding LLMs so they stop making stuff up -- an important unsolved problem.
The public repo is at https://github.com/gabegrand/world-models but the code necessary for replicating results has not been published yet.
The volume of interesting new research being done on LLMs continues to amaze me.
We sure live in interesting times!
---
PS. If any of the authors are around, please feel free to point out any errors in my understanding.
Re: From word models to world models
#3It's a surprise to see a paper actually try to solve the problem of modelling thought via language.
Nevertheless, it begins with far too many hedges:
> By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking
There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling from similar text which is distributed so-as-to-express a thought by some agent; Hyp2 -- it has the capacity to form that thought.
It is absolutely trivial to show Hyp2 is false:
> Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain.
Indeed: because there're no relevant prior cases to sample from in that case.
> These issues make it difficult to evaluate whether LLMs have acquired cognitive capacities such as social reasoning and theory of mind
It doesnt. It's trivial: the disproof lies one sentence above. Its just that many don't like the answer. Such capacities survive trivial permutations -- LLMs do not. So Hypothesis-2 is clearly false.
Re: From word models to world models
#4The level of understanding of the problem that this paper expresses is extraordianry in my reading of this field --- it's a genuinely amazing synthesis.
> How could the common-sense background knowledge needed for dynamic world model synthesis be represented, even in principle? Modern game engines may provide important clues.
This has often been my starting point in modelling the difference between a model-of-pixels vs. a world model. Any given video game session can be "replayed" by a model of its pixels: but you cannot play the game with such a model. It does not represent the causal laws of the game.
Even if you had all possible games you could not resolve between player-caused and world-caused frames.
> A key question is how to model this capability. How do minds craft bespoke world models on the fly, drawing in just enough of our knowledge about the world to answer the questions of interest?
This requires a body: the relevant information missing is causal, and the body resolves P(A|B) and P(A|B->A) by making bodily actions interpreted as necessarily causal.
In the case of video games, since we hold the controller, we resolve P(EnemyDead|EnemyHit) vs. P(EnemyDead| (ButtonPress ->) EnemyHit -> EnemyDead)
Re: From word models to world models
#5I doubt that word models can lead to world models. To quote Yann LeCun:
"The vast majority of our knowledge, skills, and thoughts are not verbalizable. That's one reason machines will never acquire common sense solely by reading text."
Re: From word models to world models
#6It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…
I don't think you really disproved anything. You're just saying another hypothesis. Often, LLMs produce impressive results on domains that aren't in the training set.
Re: From word models to world models
#7It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…
>It is absolutely trivial to show Hyp2 is false
No it's not
> Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain.
>Indeed: because there're no relevant prior cases to sample from in that case.
That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolutionary biology (some still look absurd) and that don't speak to any lack or presence of intelligence or complex thought. Not sure why it's so hard to grasp that LLMs are bound to have odd failure modes regardless of the above.
and trivial here is relative. In my experience, "trivial" often turns out to be trivial in the way a person may not pay close attention to and be similarly tricked.
For instance, GPT-4 might solve a classic puzzle correctly then fail the same puzzle subtlety changed. I've found more often than not, simply changing names of variables in the puzzle to something completely different can get it to solve the changed puzzle. It takes memory shortcuts but can be pulled out of that. LLMs have failure modes that look like human failure modes too.
Re: From word models to world models
#8It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…
>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolut…
The "failure modes" in humans do not show we lack the capacity.
Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object.
> I've found more often than not, simply changing names of variables
Yes, lol --- why do you think that is?
Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset that can be sampled from by the LLM. Showing Hyp1 to be the case.
P(Hyp1| ChangeNameMakesDifference) >>>>>> P(Hyp2|ChangeNameMakesDifference)
To such a degree that the latter is vanishingly close to zero.
Re: From word models to world models
#9Earlier quoted context omitted.
>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolut…
The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object . > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset…
>The "failure modes" in humans do not show we lack the capacity.
Then they don't in LLMs too
>Yes, lol --- why do you think that is?
Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of there too but that doesn't work as often.
Re: From word models to world models
#10It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…
If it's "absolutely trivial" to show that LLMs don't have the capacity to form thought, then please publish a paper proving that. So all the "stupid" people studying LLMs that can't come up with such trivial proofs can move on to other stuff.