Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evaluating the world model implicit in a generative model

arxiv.org

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Evaluating the world model implicit in a generative model

#21

Post by shanusmagnus »

mistercow wrote: post #17
HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones). I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process. For example, we model hurricane…

Brilliant analogy.

And we can imagine that, in a sci-fi world where some super-being could act on a scale that would allow it to perturb the world in a fashion amenable to causing hurricanes, the hurricane model could be substantially augmented, for the same reason motor babbling in an infant leads to fluid motion as a child.

What has been a revelation to me is how, even peering through this dark glass, titanic amounts of data allow quite useful world models to emerge, even if they're super limited -- a type of "bitter lesson" that suggests we're only at the beginning of what's possible.

I expect robotics + LLM to drive the next big breakthroughs, perhaps w/ virtual worlds [1] as an intermediate step.

[1] https://minedojo.org/

Re: Evaluating the world model implicit in a generative model

#22

Post by shanusmagnus »

plra wrote: post #15

Really cool results. I'd love to see some human baselines for, say, NYC cabbies or regular Manhattanites, though. I'm sure my world model is "incoherent" vis-a-vis these metrics as well, but I'm not sure what degree of coherence I should be excited about.

Makes me think of an interesting related question: how aware are we, normally, of our incoherence? What's the phenomenology of that? Hmm.

Re: Evaluating the world model implicit in a generative model

#23

Post by stonemetal12 »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

People around here like to say "The map isn't the territory". If we are talking about the physical world, then language is a map not the territory, and not a detailed one either, an LLM trained on it is a second order map.

If we consider the territory to be human intelligence, then language is still a map but it is a much more detailed map. Thus an LLM trained on it becomes a more interesting second order map.

Re: Evaluating the world model implicit in a generative model

#24

Post by seydor »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

Animals could well use an autoregressive model to predict the outcomes of their actions on their perceptions. It's not like we run math in out everyday actions (it would take too long).

Perhaps thats why we can easily communicate those predictions as words

Re: Evaluating the world model implicit in a generative model

#25

Post by Jerrrrrrry »

zxexz wrote: post #8
zxexz wrote: post #7

Earlier quoted context omitted.

Check out PyKEEN [0] and go wild. I like to train a bunch of random models and "overfit" them to the extreme (in my mind overfitting them is the point for this task, you want dense, compressed knowledge). Resize the input and output embeddings of an existing pretrained (but small) LLM (input only necessary if you're adding extra metadata on input, but make sure you untie input/output weights). You can add a linear la…

Sorry if that was ridiculously vague. I don't know a ton about the state of the art, and I'm really not sure there is one - the papers just seem to get more terminology-dense and the research mostly just seems to end up developing new terminology. My grug-brained philosophy is just to make models small enough you can just shove things in and iterate quick enough in colab or a locally hosted notebook with access to a…

Thank you for your comments (good further reading terms), and your open invitation for continued inquiry.

The "fomo" / deja vu / impending doom / incipient shift in the Overton window regarding meta-architecture for AI/ML capabilities and risks is so now glaring obvious of an elephant in the room it is nearly catatonic to some.

https://www.youtube.com/watch?v=2ziuPUeewK0

Re: Evaluating the world model implicit in a generative model

#27

Post by HarHarVeryFunny »

mistercow wrote: post #17
HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones). I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process. For example, we model hurricane…

You can model a generative process, but it's necessarily an auto-regressive generative process, not the same as the originating generative process which is based on the external world.

Human language, and other actions, exist on a range from almost auto-regressive (generating a stock/practiced phrase such as "have a nice day") to highly interactive ones. An auto-regressive model is obviously going to have more success modelling an auto-regressive generative process.

Weather prediction is really a good case of the limitation of auto-regressive models, as well as models that don't accurately reflect the inputs to the process you are attempting to predict. "There's a low pressure front coming in, so the weather will be X, same as last time", works some of the time. A crude physical weather model based on limited data points, such as weather balloon inputs, or satellite observation of hurricanes, also works some of the time. But of course these models are sometimes hopelessly wrong too.

My real point wasn't about the lack of sensory data, even though this does force a purely auto-regressive (i.e. wrong) model, but rather about the difference between a passive model (such as weather prediction), and an interactive one.

Re: Evaluating the world model implicit in a generative model

#28

Post by _yb2s »

Really glad to see some academic research on this- it was quite obvious from interacting with LLMs that they form a world model and can, e.g. simulate simple physics experiments correctly that are not in the training set. I found it very frustrating to see people repeating the idea that “it can never do x” because it lacks a world model. Predicting text that represents events in the world requires modeling that world. Just because you can find examples where the predictions of a certain model are bad does not imply no model at all. At the limit of prediction becoming as good as theoretically possible given the input data and model size restrictions, the model also becomes as accurate and complete as possible. This process is formally described by the Solomonoff Induction theory.

Re: Evaluating the world model implicit in a generative model

#29

Post by slashdave »

_yb2s wrote: post #28

Really glad to see some academic research on this- it was quite obvious from interacting with LLMs that they form a world model and can, e.g. simulate simple physics experiments correctly that are not in the training set. I found it very frustrating to see people repeating the idea that “it can never do x” because it lacks a world model. Predicting text that represents events in the world requires modeling that world…

> At the limit of prediction becoming as good as theoretically possible given the input data and model size restrictions

You are treading on delicate ground here. Why do you believe that sequence models are capable of reaching theoretical maximums?

Re: Evaluating the world model implicit in a generative model

#30

Post by slashdave »

Most of you probably know someone with a poor sense of direction (or may be yourself). From my experience, such people navigate primarily (or solely) by landmarks. This makes me wonder if the damaged maps shown in the paper are similar to the "world model" belonging to a directionally challenged person.
Post reply on HN