Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evaluating the world model implicit in a generative model

arxiv.org

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Evaluating the world model implicit in a generative model

#31

Post by slashdave »

mistercow wrote: post #17
HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones). I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process. For example, we model hurricane…

Indeed. If you provided a talented individual with a sufficient quantity and variety of video streams of travels in a city (like New York), that person would be able to draw you a map.

Re: Evaluating the world model implicit in a generative model

#32

Post by dsubburam »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

> The "world model" of a human, or any other animal, is built pursuant to predicting the environment

What do you make of Immanuel Kant's claim that all thinking has as a basis the presumption of the "Categories"--fundamental concepts like quantity, quality and causality[1]. Do LLMs need to develop a deep understanding of these?

[1] https://plato.stanford.edu/entries/categories/#KanCon

Re: Evaluating the world model implicit in a generative model

#33

Post by fragmede »

IshKebab wrote: post #13
fragmede wrote: post #2

Wrong as it is, I'm impressed they were able to get any maps out of their LLM that look vaguely cohesive. The shortest path map has bits of streets downtown and around Central Park that aren't totally red, and Central Park itself is clear on all 3 maps. They used eight A100s, but don't say how long it took to train their LLM. It would be interesting to know the wall clock time they spent. Their dataset is, relatively…

It's a bit unclear what the map visualisations are showing to me, but I don't think your interpretation is correct. They even say: > Our evaluation methods reveal they are very far from recovering the true street map of New York City. As a visualization, we use graph reconstruction techniques to recover each model’s implicit street map of New York City. The resulting map bears little resemblance to the actual streets…

My read of

> Edges exit nodes in their specified cardinal direction. In the zoomed-in images, edges belonging to the true graph are black and false edges added by the reconstruction algorithm are red.

is that the model output edges, valid ones were then colored black and bad ones colored red. But it's a bit unclear so you could be right.

Re: Evaluating the world model implicit in a generative model

#34

Post by anon291 »

Jerrrrrrry wrote: post #26

Once your model and map get larger than the thing it is modeling/mapping, then what? Let us hope the Pigeonhole principle isn't flawed, else we can find ourselves batteries in the Matrix.

In the paper 'Hopfield networks are all you need', they calculate the total number of things able to be 'stored' in the attention layers, and it's exponential in the number of parameters. So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good.

From a technical perspective, this is due to the softmax activation function that causes high degrees of separation between memory points.

Re: Evaluating the world model implicit in a generative model

#35

Post by machiaweliczny »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

But if you squint then sensory actions and reactions are also sequential tokens. Even reactions can be encoded alongside input as action tokens and as single token stream. Anyone tried sth like this?

Re: Evaluating the world model implicit in a generative model

#36

Post by Jerrrrrrry »

anon291 wrote: post #34
Jerrrrrrry wrote: post #26

Once your model and map get larger than the thing it is modeling/mapping, then what? Let us hope the Pigeonhole principle isn't flawed, else we can find ourselves batteries in the Matrix.

In the paper 'Hopfield networks are all you need', they calculate the total number of things able to be 'stored' in the attention layers, and it's exponential in the number of parameters. So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good. From a technical perspective, this is due to the softmax activation function that causes high degrees of separatio…

   > So essentially, you can store more 'ideas' in an LLM than there are particles in the universe. I think we'll be good.
If it can compress humanities knowledge corpus to we are fukt

Re: Evaluating the world model implicit in a generative model

#37

Post by gwern »

isaacfrond wrote: post #4

I think there is a philosophical angle to this. I mean, my world map was constructed by chance interactions with the real world. Does this mean that the my world map is a close to the real world map, as their NN's map is to Manhattan? Is my world map full of non-existent streets, exits that are at the wrong place, etc. The NN map of Manhattan works almost 100% correctly when used for normal navigation but breaks apar…

One of the things about offline imitation learning like OP or LLMs in general is that the more important the error in their world model, the faster it'll correct itself. If you think you can teleport across a river, you'll make & execute plans which exploit that fact first thing to save a lot of time - and then immediately hit the large errors in that plan and observe a new trajectory which refutes an entire set of errors in your world model. And then you retrain and now the world model is that much more accurate. The new world model still contains errors, and then you may try to exploit those too right away, and then you'll fix those too. So the errors get corrected when you're able to execute online with on-policy actions. The errors which never turn out to be relevant won't get fixed quickly, but then, why do you care?

Re: Evaluating the world model implicit in a generative model

#38

Post by nerdponx »

HarHarVeryFunny wrote: post #27
mistercow wrote: post #17

Earlier quoted context omitted.

> It can't model the generative process of the humans who created those training set samples because that generative process has different inputs - sensory ones (in addition to auto-regressive ones). I think that’s too strong a statement. I would say that it’s very constrained in its ability to model that, but not having access to the same inputs doesn’t mean you can’t model a process. For example, we model hurricane…

You can model a generative process, but it's necessarily an auto-regressive generative process, not the same as the originating generative process which is based on the external world. Human language, and other actions, exist on a range from almost auto-regressive (generating a stock/practiced phrase such as "have a nice day") to highly interactive ones. An auto-regressive model is obviously going to have more succes…

The whole innovation of GPT and LLMs in general is that an autoregressive model can make alarmingly good next-token predictions with the right inductive bias, a large number of parameters, a long context window, and a huge training set.

It turns out that human communication is quite a lot more "autoregressive" than people assumed it was up until now. And that includes some level of reasoning capability, arising out of a kind of brute force pattern matching. It has limits, of course, but it's amazing that it works as well as it does.

Re: Evaluating the world model implicit in a generative model

#39

Post by madaxe_again »

HarHarVeryFunny wrote: post #20
madaxe_again wrote: post #19

Earlier quoted context omitted.

You say this, yet people such as Helen Keller suggest that a full sensorium is not necessary to be a full human. She had some grasp of the idea of colour, of sound, and could use the words around them appropriately - yet had no firsthand experience of either. Is it really so different? I think “we” each comprise a number of models, language being just one of them - however an extremely powerful one, as it allows the…

It's not about the necessity of specific sensory inputs, but rather about the difference in type of model that will be built when the goal is passive, and auto-regressive, as opposed to when the goal is interactive. In the passive/auto-regressive case you just need to model predictive contexts. In the interactive case you need to model dynamical behaviors.

I don’t know that I see the difference - but I suppose we’re getting into Brains In Vats territory. In my view (well, Baudrillard’s view, but who’s counting?) a perfect description of a thing is as good as the thing itself, and we in fact interact with our semantic description of reality, rather than with raw reality itself - the latter, when it manifests in humans, results in vast cognitive dysfunction - Sachs wrote somewhat in the topic of unfiltered sensorium and the impact on the ability to operate in the world.

So yeah. I think what these models do and what we do is more similar than we might realise.

Re: Evaluating the world model implicit in a generative model

#40

Post by _yb2s »

slashdave wrote: post #29
_yb2s wrote: post #28

Really glad to see some academic research on this- it was quite obvious from interacting with LLMs that they form a world model and can, e.g. simulate simple physics experiments correctly that are not in the training set. I found it very frustrating to see people repeating the idea that “it can never do x” because it lacks a world model. Predicting text that represents events in the world requires modeling that world…

> At the limit of prediction becoming as good as theoretically possible given the input data and model size restrictions You are treading on delicate ground here. Why do you believe that sequence models are capable of reaching theoretical maximums?

I do not think any real systems can ever achieve theoretically perfect Solomonoff Induction- only that increasingly good AI systems can be thought of as increasingly good approximations of this process. I do not know if any particular modeling approach has a fundamental dead end that limits its potential or not. However, my main point is that people claiming that they are certain of a particular fundamental limitation are mistaken. Current LLMs aren’t very intelligent, yet can already do specific things that people like Noam Chomsky have argued are fundamentally theoretically impossible for them to ever do.
Post reply on HN