Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Evaluating the world model implicit in a generative model

arxiv.org

Post reply on HN
Print view View original

41–47 of 47 posts

Re: Evaluating the world model implicit in a generative model

#41

Post by slashdave »

_yb2s wrote: post #40
slashdave wrote: post #29

Earlier quoted context omitted.

> At the limit of prediction becoming as good as theoretically possible given the input data and model size restrictions You are treading on delicate ground here. Why do you believe that sequence models are capable of reaching theoretical maximums?

I do not think any real systems can ever achieve theoretically perfect Solomonoff Induction- only that increasingly good AI systems can be thought of as increasingly good approximations of this process. I do not know if any particular modeling approach has a fundamental dead end that limits its potential or not. However, my main point is that people claiming that they are certain of a particular fundamental limitatio…

> However, my main point is that people claiming that they are certain of a particular fundamental limitation are mistaken.

No, they are correct. The architecture, by design and construction, is limited. This is simple math.

Re: Evaluating the world model implicit in a generative model

#42

Post by westurner »

dsubburam wrote: post #32
HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

> The "world model" of a human, or any other animal, is built pursuant to predicting the environment What do you make of Immanuel Kant's claim that all thinking has as a basis the presumption of the "Categories"--fundamental concepts like quantity, quality and causality[1]. Do LLMs need to develop a deep understanding of these? [1] https://plato.stanford.edu/entries/categories/#KanCon

Embodied cognition implies that we understand our world in terms of embodied metaphor "categories".

LLMs don't reason, they emulate. RLHF could cause an LLM to discard text that doesn't look like reasoning according to the words in the response, but that's still not reasoning or inference.

"LLMs cannot find reasoning errors, but can correct them" https://news.ycombinator.com/item?id=38353285

Conceptual metaphor: https://en.wikipedia.org/wiki/Conceptual_metaphor

Embodied cognition: https://en.wikipedia.org/wiki/Embodied_cognition

Clean language: https://en.wikipedia.org/wiki/Clean_language

Given human embodied cognition as the basis for LLM training data, there are bound to be weird outputs about bodies from robot LLMs.

Re: Evaluating the world model implicit in a generative model

#43

Post by RaftPeople »

machiaweliczny wrote: post #35
HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

But if you squint then sensory actions and reactions are also sequential tokens. Even reactions can be encoded alongside input as action tokens and as single token stream. Anyone tried sth like this?

> But if you squint then sensory actions and reactions are also sequential tokens

I'm not sure you could model it that way.

Animal brains don't necessarily just react to sensory input, they frequently have already predicted the next state based on previous state and learning/experience, and not just in a simple sequential manner but at many different levels of patterns involved simultaneously (local immed action vs actions part of larger structure of behavior), etc.

Sensory input is compared to predicted state and differences are incorporated into the flow.

The key thing is our brains are modeling and simulating the world around us and it's future state (modeling the physical world as well as the abstract world of what other animals are thinking). It's not clear that LLM's are doing that (my assumption is that they are not doing any of that, and until we build systems that do that, we won't be moving towards the kind of flexible and adaptable control our brains have).

Edit: I just read the rest of the parent post that said basically the same thing, was skimming so missed it.

Re: Evaluating the world model implicit in a generative model

#44

Post by _yb2s »

slashdave wrote: post #41
_yb2s wrote: post #40

Earlier quoted context omitted.

I do not think any real systems can ever achieve theoretically perfect Solomonoff Induction- only that increasingly good AI systems can be thought of as increasingly good approximations of this process. I do not know if any particular modeling approach has a fundamental dead end that limits its potential or not. However, my main point is that people claiming that they are certain of a particular fundamental limitatio…

> However, my main point is that people claiming that they are certain of a particular fundamental limitation are mistaken. No, they are correct. The architecture, by design and construction, is limited. This is simple math.

Limited how exactly? What limitation are you talking about, and what math proves it?

Re: Evaluating the world model implicit in a generative model

#45

Post by comfysocks »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

It seems to me that the human authors of the training text are the ones who have created the “world model”, and have encoded it into written language. The llm transcodes this model into word embedding vector space. I think most people can recognize a high dimensional vector space as a reasonable foundation for a mathematical “model”. The humans are the ones who have interacted with the world and have perceived its workings. The llm only interacts with the human’s language model. Some credit must be given to the humans modellers for the unreasonable effectiveness of the llm.

Re: Evaluating the world model implicit in a generative model

#46

Post by ElevenLathe »

HarHarVeryFunny wrote: post #16

An LLM necessarily has to create some sort of internal "model" / representations pursuant to its "predict next word" training goal, given the depth and sophistication of context recognition needed to to well. This isn't an N-gram model restricted to just looking at surface word sequences. However, the question should be what sort of internal "model" has it built? It seems fashionable to refer to this as a "world mode…

We can't see neutrons either, but we have built various models of them based on indirect observations.

Re: Evaluating the world model implicit in a generative model

#47

Post by HarHarVeryFunny »

nerdponx wrote: post #38
HarHarVeryFunny wrote: post #27

Earlier quoted context omitted.

You can model a generative process, but it's necessarily an auto-regressive generative process, not the same as the originating generative process which is based on the external world. Human language, and other actions, exist on a range from almost auto-regressive (generating a stock/practiced phrase such as "have a nice day") to highly interactive ones. An auto-regressive model is obviously going to have more succes…

The whole innovation of GPT and LLMs in general is that an autoregressive model can make alarmingly good next-token predictions with the right inductive bias, a large number of parameters, a long context window, and a huge training set. It turns out that human communication is quite a lot more "autoregressive" than people assumed it was up until now. And that includes some level of reasoning capability, arising out o…

It is amazing, and interesting.

Although I used the word myself, I'm not sure that "autoregressive" is quite the right word to describe how LLMs work, or our brains. Maybe better to just call both "predictive". In both cases the predictive inputs include the sequence itself (or selected parts of it, at varying depths of representation), but also global knowledge, both factual and procedural (HOW to represent the sequence). In the case of our brain there are also many more inputs that may be used such as sensory ones (passive observations, or action feedback), emotional state, etc.

Regardless of what predictive inputs are available to LLMs vs brains, it does seem that in a lot of cases the more constrained inputs of an LLM don't prevent it from sounding very human like (not surprising at some level given the training goal), and an LLM chat window does create a "level playing field" (i.e. impoverished input setting for the human) where each side only sees the other as a stream of text. Maybe in this setting, the human, when not reasoning, really isn't bringing much more predictive machinery to the table than the LLM/transformer!

Notwithstanding the predictive nature of LLMs, I can't help but also see them just as expert systems of sorts, albeit ones that have derived their own rules (much pertaining to language) rather than being given them. This view better matches their nature as fixed repositories of knowledge, brittle where rules are missing, as opposed to something more brain-like and intelligent, capable of continual learning.

Post reply on HN