Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From word models to world models

arxiv.org

Post reply on HN
Print view View original

11–20 of 119 posts

Re: From word models to world models

#11

Post by fiso64 »

mjburgess wrote: post #3

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…

Don't try to ham-fist scientific sounding wording into your (very unscientific) argument. This is not a disproof of anything because you failed to define what it means to have the ability to form rational thoughts. With a definition, you would then wanna prove this for humans as a sanity check: Do we never make stupid mistakes? Ok, we make fewer of those than LLMs. Then what is the threshold for accuracy after which you consider a system to be intelligent? Do all humans pass that threshold, or do kids or people with a lower than average IQ fail?

Re: From word models to world models

#12

Post by mdp2021 »

fiso64 wrote: post #11
mjburgess wrote: post #3

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…

Don't try to ham-fist scientific sounding wording into your (very unscientific) argument. This is not a disproof of anything because you failed to define what it means to have the ability to form rational thoughts. With a definition, you would then wanna prove this for humans as a sanity check: Do we never make stupid mistakes? Ok, we make fewer of those than LLMs. Then what is the threshold for accuracy after which…

> humans

There is intelligent thought and action, and there is unintelligent thought and action. Intelligent is that "which checked" (intus-legere); the other, the """impulsive""", is not.

Re: From word models to world models

#13

Post by mjburgess »

fiso64 wrote: post #11
mjburgess wrote: post #3

It's a surprise to see a paper actually try to solve the problem of modelling thought via language. Nevertheless, it begins with far too many hedges: > By scaling to even larger datasets and neural networks, LLMs appeared to learn not only the structure of language, but capacities for some kinds of thinking There's two hypotheses for how LLMs generate apparently "thought-expressing" outputs: Hyp1 -- it's sampling fro…

Don't try to ham-fist scientific sounding wording into your (very unscientific) argument. This is not a disproof of anything because you failed to define what it means to have the ability to form rational thoughts. With a definition, you would then wanna prove this for humans as a sanity check: Do we never make stupid mistakes? Ok, we make fewer of those than LLMs. Then what is the threshold for accuracy after which…

This entire paper is written as a disproof of the distributional hypothesis. If you want to understand why it's a profoundly unhelpful pseudoscientific idea, this paper is a good start.

The test for a capacity C in a system1 has nothing to do with proxy measures of that capacity in system2.

The capacity for an oven to cook food may be measured by how much smoke it lets of when burning -- but no amount of "smoke" establishes that a dry ice machine can cook.

This type of "engineering thinking" is pseudoscience.

Re: From word models to world models

#14

Post by FrustratedMonky »

antiquark wrote: post #5

I doubt that word models can lead to world models. To quote Yann LeCun: "The vast majority of our knowledge, skills, and thoughts are not verbalizable. That's one reason machines will never acquire common sense solely by reading text." https://twitter.com/ylecun/status/1368235803147649028

>"solely by reading text".

Of course, that does leave the door Open, that when these models are put in a physical real body, a robot, and have to interact with the world, then maybe they can gain that "common sense".

This doesn't mean a silicon based AI can't become conscious of skills that are hard to verbalize. Just that they don't yet have all the same inputs that we have. And when they do, and they have internal thoughts, they will have the same difficulty verbalizing them that we do.

Re: From word models to world models

#15

Post by lgessler »

mjburgess wrote: post #8
famouswaffles wrote: post #7

Earlier quoted context omitted.

>It is absolutely trivial to show Hyp2 is false No it's not > Current LLMs can produce impressive results on a set of linguistic inputs and then fail completely on others that make trivial alterations to the same underlying domain. >Indeed: because there're no relevant prior cases to sample from in that case. That's not what that tells us. Humans have weird failure modes that look absurd outside the context of evolut…

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object . > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset…

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define that.

Re: From word models to world models

#16

Post by mjburgess »

famouswaffles wrote: post #9
mjburgess wrote: post #8

Earlier quoted context omitted.

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object . > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset…

>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of th…

> Then they don't in LLMs too

LLMs don't get drunk .

If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not.

They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.).

Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

But these are not the kinds of failures experienced with those who have some capacity, eg., for counter-factual reasoning about their environment's physics.

In those people it is environmental illusion and cognitive impairment -- not trivial permutations of phrasing which lead to catastrophic loss of apparent understanding.

Cognitive impairment = reasoning machine is broken

Environmental illusion = data is ambigious and actions cannto resolve it

These "failure modes" are expected if you actually have the relevant capacity.

Re: From word models to world models

#17

Post by mjburgess »

lgessler wrote: post #15
mjburgess wrote: post #8

Earlier quoted context omitted.

The "failure modes" in humans do not show we lack the capacity. Eg., do you have capacity to reason about physics? Well if you're extremely drunk, less so. But not if I permute the name of the object . > I've found more often than not, simply changing names of variables Yes, lol --- why do you think that is? Because in the digitised dataset of "everything ever written" those names correspond to places in that dataset…

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define tha…

I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis.

I am using science, ie., abduction, to compare a class of hypotheses.

P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

is much much much lower than,

P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

Re: From word models to world models

#18

Post by lgessler »

mjburgess wrote: post #17
lgessler wrote: post #15

Earlier quoted context omitted.

This is a false dichotomy. It's not the case that models are truly capable of reasoning if and only if they are insensitive to irrelevant perturbations to input. In other words, the mere fact that sensitivity to names sometimes causes significant degradations in model performance doesn't mean that we've observed models are incapable of anything we might call "reasoning"—leaving aside the matter of how we'd define tha…

I didnt say "if and only if" -- this is a conceptual analysis condition which applies only under deductive analysis. I am using science, ie., abduction, to compare a class of hypotheses. P(CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases) is much much much lower than, P(-CapacityToThink| DegradingPermutations, ModelDrawsFromHistoricalCases)

Idk, that isn't the sense I got from "It is absolutely trivial to show Hyp2 is false", but sure, I agree with you that this evidence certainly ought to tip the scales one way and not the other.

Re: From word models to world models

#19

Post by famouswaffles »

mjburgess wrote: post #16
famouswaffles wrote: post #9

Earlier quoted context omitted.

>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of th…

> Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial l…

>Insofar as a human cannot answer questions under trivial linguistic permutations then they too do not understand the domain.

alright let me humor you for a bit. Lets start with some solid examples of GPT-4 failing this "trivial linguistic permutation" then ?

Re: From word models to world models

#20

Post by moffkalast »

mjburgess wrote: post #16
famouswaffles wrote: post #9

Earlier quoted context omitted.

>The "failure modes" in humans do not show we lack the capacity. Then they don't in LLMs too >Yes, lol --- why do you think that is? Being able to solve a changed common puzzle but also with different names than it would ever see in training is not an indication of a lack of ability lol. and changing names isn't the only way to get it out of memory, just the easiest/most straightforward. You can converse it out of th…

> Then they don't in LLMs too LLMs don't get drunk . If a child answers questions from a book of answers then they'll appear to understand the domain insofar as those questions appear. They do not. They will fail to answer questions under, eg., permutations of words (say, a question asks about "norepinephrine" but the book only contains "noradrenaline" etc.). Insofar as a human cannot answer questions under trivial l…

> LLMs don't get drunk .

Well actually they sort of can...

https://www.reddit.com/r/LocalLLaMA/comments/13vv941/tempera...

Post reply on HN