Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From word models to world models

arxiv.org

Post reply on HN
Print view View original

51–60 of 119 posts

Re: From word models to world models

#51

Post by mjburgess »

jbay808 wrote: post #45
mjburgess wrote: post #39

Earlier quoted context omitted.

There are an infinite number of sentences which describe what "being red" is, most of them have never been written. Redness is not in the structure of those sentences. And there will always be an infinity of sentences which are True but cannot be infered by an LLM -- but can be so, trivially, by a person acquainted with redness. In any case, I'd need more time than I have at the moment to seriously state Hyp1 for you…

> Redness is not in the structure of those sentences. Sure; it's in the spectrum of reflected light. (Or perhaps, the retina's trichromal responsivity). But that physical concept can be meaningfully described by sentences. It doesn't require an infinite number of them to create a coherent world-model, which can do things like predicting that a blue object will become red if it moves away from you at a high enough spe…

> that's no guarantee that the LLM will be able to compress it efficiently

Your LLM here is 600MB which is a grossly inefficient compression of the sort space.

If LLMs "learned algorithms", the best compression would be on the order of bytes.

The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it!

What do you think all those MBs are doing? They're the extraordinary cost of the "statistical shortcut" of modelling the empirical distribution of sorted numbers.

NNs exploit distributional structure in the training data to compress it --- in this case there's huge amounts of distributional structure in numbers.

I think you've misunderstood the "statistical parrot" claim to be somehow that NNs are engaged in wrote memorization... or, what?

The claim is simply that all they do is statistically approximate the empirical distribution of the training dataset structure --- and if you force interpolation, then they provide arbitrarily precise compressions of that structure.

I'm not sure what a NN which can sort numbers shows, other than the distributional structure of a sort-numbers dataset is such that a NN can compress it into 600MB...

To be clear, the "statistical parrot" claim is that the statistical distribution of the empirical dataset D = (X, y) is being approximated by the weights, W = Compress(D) -- and that this distribution fails to be a representational model of y -- because no entailments of X (other than those in D) are captured.

Whereas representational models are not confined to the distribution of historical cases, ie., I can imagine variations on X leading to any given y; and variations on y leading to any given X -- without ever having experienced either.

You're showing the system vast amounts of numbers being sorted, so it learns the distribution of that data, so it can replay those sorts.

I'm not exactly sure why you think this is a reply to the relevant claims.

Re: From word models to world models

#52

Post by lostmsu »

mjburgess wrote: post #34
lostmsu wrote: post #28

Earlier quoted context omitted.

> Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. Why? This is obviously wrong in general case. For that to be true Xp and Xc has to have no statistical relationship whatsoever, which statistically is virtually impossible.

Xp just have to be chosen such that the distribution Xc,Xp is sufficiently small in the training data -- but not that the tokens of Xp are themselves rare. So that an agent competent with tokens in X, who can construct repr of S, could do so with Xp. Consider a reference in the paper above, https://arxiv.org/pdf/2302.08399.pdf Xc = > Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label…

> just have to be chosen such that the distribution Xc,Xp is sufficiently small in the training data -- but not that the tokens of Xp are themselves rare

Great idea. Now prove you can actually choose such a distribution, lol.

Re: From word models to world models

#53

Post by [deleted] »

achrono wrote: post #38
mjburgess wrote: post #17

Earlier quoted context omitted.

I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

This might be a naive question, but here me out. Do we really know what the difference is between statistics and the capacity to think? Is "true understanding" rather a continuum of sophistication from a simple adder to Albert Einstein? My point here isn't "if it quacks like a duck...", but more so that while we are talking about intelligent apparatus we should be comparing apples to apples, and not say "this is a me…

[deleted]

Re: From word models to world models

#54

Post by mjburgess »

lostmsu wrote: post #52
mjburgess wrote: post #34

Earlier quoted context omitted.

Xp just have to be chosen such that the distribution Xc,Xp is sufficiently small in the training data -- but not that the tokens of Xp are themselves rare. So that an agent competent with tokens in X, who can construct repr of S, could do so with Xp. Consider a reference in the paper above, https://arxiv.org/pdf/2302.08399.pdf Xc = > Here is a bag filled with popcorn. There is no chocolate in the bag. Yet, the label…

> just have to be chosen such that the distribution Xc,Xp is sufficiently small in the training data -- but not that the tokens of Xp are themselves rare Great idea. Now prove you can actually choose such a distribution, lol.

I think this is easy, just make Xp sentences of the kind = "I define `randomchars()` to be this `term-in-Xc()`" and swamp the dataset with Xc.

Everything here actually just follows formally from what NNs are: they're just empirical function approximations.

It will always be the case that they just model the probabilistic structure of the dataset and not the data generating process.

Since, in language, there are discrete constraints which make P(...) = 1 or P(...) = 0 --- you can trivially produce datasets showing that it learns P(...) = mistake-you-created-deliberately and not either 0,1.

As above, the LLM switches from 95% confidence "chocolate" to 95% confidence "popcorn" with a trivial non-semantic permutation of the prompt.

The obscene issue in all this is that we know this already -- empirical function approximation of historical datasets just produces associative probabilistic models of those datasets.

Re: From word models to world models

#55

Post by jstanley »

mjburgess wrote: post #51
jbay808 wrote: post #45

Earlier quoted context omitted.

> Redness is not in the structure of those sentences. Sure; it's in the spectrum of reflected light. (Or perhaps, the retina's trichromal responsivity). But that physical concept can be meaningfully described by sentences. It doesn't require an infinite number of them to create a coherent world-model, which can do things like predicting that a blue object will become red if it moves away from you at a high enough spe…

> that's no guarantee that the LLM will be able to compress it efficiently Your LLM here is 600MB which is a grossly inefficient compression of the sort space. If LLMs "learned algorithms", the best compression would be on the order of bytes. The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it! What do you think all those MBs are doing? They're the extraordinary cost of the "stati…

> The python to generate this list is c. 1kb -- and you're using an obscene 600MB to do it!

This isn't a fair comparison. The python code to sort a list is leveraging an enormous amount of information that is stored outside the python code, whereas the GPT version basically has to do it "from scratch", and in a very convoluted computing model.

A better comparison would be "how many bits does it take to encode a configuration of NAND gates that describes a computer that can sort 127-byte lists of number 1..100?"

I'm sure it's not as much as 600 megabytes, but it'll be a lot more than the python code.

Re: From word models to world models

#56

Post by mjburgess »

cypress66 wrote: post #47
mjburgess wrote: post #22

Earlier quoted context omitted.

You may wish to read the paper above. But if you want a quick proof: 1. A thought is a representation of a situation 2. A representation generates entailments of that situation 3. Language is many-to-one translation from these representations to symbols 4. Understanding language is reversing these symbols into thoughts (ie., reprs) So, 5. If agent A understands sentence X then A forms the relevant representation of X…

> 8. Train an LLM on Xc and it's accuracy on judging Y entailed by Xp is random. This is clearly where the "proof" falls apart. Even in tasks where GPT4 struggles, it's accuracy will still be better than random. The bar of "better than random" is so low that even weak LLMs will be able to surpass it. More so, you need to prove not just a single, but that no task/domain exists for which LLMs satisfy 8. What your proof…

I just need to be able to create a split in Xc,Xp so that Xp is random. I think that's really quite easy.

If you could put ChatGPT in a loop, take some Xc prompts and permute with some non-semantic phrases ("Alice believes that... Xc ... what did Alice believe?") etc --- until you find those cases.

I imagine we will discover quite a large number of such non-semantic phrases which have this effect. Because the tokens in those phrases will, joint with Xc, be arbitrarily distributed in some historical data (distributed to our preference when finding them).

This seems just kinda basically obvious, right? Entailments are discretely constrained by semantics, and historical datasets can contain arbitrary mixtures of random distributions of syntax.

NNs only model those distributions -- and not the entailments -- which, at the very least, are extremely discrete.

Re: From word models to world models

#57

Post by pphysch »

moffkalast wrote: post #21
antiquark wrote: post #5

I doubt that word models can lead to world models. To quote Yann LeCun: "The vast majority of our knowledge, skills, and thoughts are not verbalizable. That's one reason machines will never acquire common sense solely by reading text." https://twitter.com/ylecun/status/1368235803147649028

That just seems like an unfounded hot take. Of course we can explain most of our knowledge, skills, and thoughts in words, that's how we don't lose everything when the next generation comes around lol. It's the core reason we're different from animals. Now sure you can't describe qualia, but that's basically a subjective artefact of how we sense the world and (to add another unfounded hot take) likely not critical to…

> Of course we can explain most of our knowledge, skills, and thoughts in words, that's how we don't lose everything when the next generation comes around lol.

I would wager if you put a newborn human to be raised in the absence of any physical human contact, but somehow taught them to read/write, and gave them access to a universal corpus (text only, no audio/video), or heck, even internet access with `curl`, and lastly dropped them into the "real world" at age 25, they would be utterly incapable of performing, say, a basic service job at a restaurant.

Words help us symbolize and reason about our sense experiences, but they are not a substitute for them.

Re: From word models to world models

#58

Post by dimatura »

This is really interesting. The title is referencing the "Language of Thought" hypothesis from early cognitive psychology, that posited thought consisted of symbol manipulation akin to computer programs. The same idea was behind was also what is often referred to GOFAI. But the idea has largely fallen out of fashion in both psychology and AI. There's a twist here in the "probabilistic" part, and of course the surprising success of LLMs makes this a more compelling idea than it would've been only a couple of years ago. And there's also an acknowledgement of the need for some kind of sensorimotor grounding as well. Pretty cool!

Re: From word models to world models

#59

Post by stevenhuang »

antiquark wrote: post #5

I doubt that word models can lead to world models. To quote Yann LeCun: "The vast majority of our knowledge, skills, and thoughts are not verbalizable. That's one reason machines will never acquire common sense solely by reading text." https://twitter.com/ylecun/status/1368235803147649028

Yann LeCun has a vested interest in downplaying LLM emergent abilities.

His research at meta is in the analytic approach to machine learning. As result he is very unabashed in expressing distaste of ML approaches that don't align with his research.

Really, there is no larger sore loser than LeCun in internalizing the bitter lesson. Quoting him without this context is being deliberately misleading.

Re: From word models to world models

#60

Post by stevenhuang »

antisthenes wrote: post #48

World modeling is impossible without sensory input. You need constant modeling of touch/smell/vision/temperature, etc. These senses give us an actual understanding of the physical world and drive our behavior in a way that pure language will never be able to.

A facsimile of sufficient equivalence to the world models we derive from our 5 senses may be approached through derivation of descriptive language only.

"sufficient equivalence" is important because sure it may not _really_ know the color of red or the qualia of being, but if for all intents and purposes the LLM's internal model provides predictive power and answers correctly as if it does have a world model, then what is the difference?

Post reply on HN