Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Can language models serve as text-based world simulators?

arxiv.org

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Can language models serve as text-based world simulators?

#21

Post by TillE »

sbotzek wrote: post #15

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task. One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. Cha…

I think the best thing to do in a scenario like that is standard procgen where you randomly, logically generate the rooms with a bunch of descriptive tags, and then LLM-ify the room descriptions, with some context for what the world/area is supposed to be like.

Re: Can language models serve as text-based world simulators?

#22

Post by Waterluvian »

sbotzek wrote: post #15

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task. One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. Cha…

I think the issue might be that people like to throw the tool at the whole problem when it’s not the right tool for a lot of it.

Don’t use LLMs for logic. Use it for colour and flavour. Generate your own layout, populate the rooms in a manner befitting the context, difficulty, story arc, and then whenever a user makes an action, update the state then pass to the LLM the state and ask it to describe the room and make a few other smaller decisions.

Re: Can language models serve as text-based world simulators?

#24

Post by adastra22 »

TillE wrote: post #21
sbotzek wrote: post #15

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task. One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. Cha…

I think the best thing to do in a scenario like that is standard procgen where you randomly, logically generate the rooms with a bunch of descriptive tags, and then LLM-ify the room descriptions, with some context for what the world/area is supposed to be like.

It can even dynamically extend by using prompts to generate new rooms as the user enters them, but keeping track of generated world outside of the LLM.

Which is the same way nearly all procedurally generated games work.

Re: Can language models serve as text-based world simulators?

#25

Post by Folcon »

danjc wrote: post #18
sbotzek wrote: post #15

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task. One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. Cha…

Somewhat related, Yann LeCun posted a few months back about how many concepts aren't understood through language and therefore can't be modeled through it (which is why LLM's are terrible with things like position and direction). https://x.com/ylecun/status/1768353714794901530?s=46

    To people who claim that "thinking and reasoning require language", here is a problem:
    Imagine standing at the North Pole of the Earth.
    Walk in any direction, in a straight line, for 1 km.
    Now turn 90 degrees to the left.
    Walk for as long as it takes to pass your starting point.
    Have you walked:
    1. More than 2xPi km
    2. Exactly 2xPi km
    3. Less than 2xPi km
    4. I never came close to my starting point.

    Think about how you tried to answer this question and tell us whether it was based on language.
Just quoting this here in case anything happens to the tweet...

I agree with this, however I have one tiny nitpick, feel free to tell me if you think I'm wrong or being overly nitpicky, but the knowledge of the situation in which the phenomenon that he's describing occurs, I learned about entirely from language. So my ability to answer the question is based on that knowledge.

I'm aware that the reasoning problem itself doesn't utilise it, but a position and direction system in and of itself arguably also suffers from being insufficient.

I suppose I'm wondering if "setup" counts as needing language model?

Re: Can language models serve as text-based world simulators?

#27

Post by maxglute »

danjc wrote: post #18
sbotzek wrote: post #15

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task. One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. Cha…

Somewhat related, Yann LeCun posted a few months back about how many concepts aren't understood through language and therefore can't be modeled through it (which is why LLM's are terrible with things like position and direction). https://x.com/ylecun/status/1768353714794901530?s=46

This interaction down the chain is interesting:

Q: yann do you have an internal monologue?

A: Not that I know of.

Re: Can language models serve as text-based world simulators?

#28

Post by dragonwriter »

Folcon wrote: post #25
danjc wrote: post #18

Earlier quoted context omitted.

Somewhat related, Yann LeCun posted a few months back about how many concepts aren't understood through language and therefore can't be modeled through it (which is why LLM's are terrible with things like position and direction). https://x.com/ylecun/status/1768353714794901530?s=46

To people who claim that "thinking and reasoning require language", here is a problem: Imagine standing at the North Pole of the Earth. Walk in any direction, in a straight line, for 1 km. Now turn 90 degrees to the left. Walk for as long as it takes to pass your starting point. Have you walked: 1. More than 2xPi km 2. Exactly 2xPi km 3. Less than 2xPi km 4. I never came close to my starting point. Think about how yo…

I don't think the example is a good rebuttal of "thinking and reasoning require language".

It may be a decent challenge, probably still not an actual rebuttal, of "language is sufficient for all thinking and reasoning", but "X is required for Y" and "X is sufficient for everything encompassed by Y" are very different claims.

Re: Can language models serve as text-based world simulators?

#29

Post by carabiner »

Too much data are locked up in non-public sources that still affect the world. You will not find complete engineering analysis reports for the Ford F-150 engine on the internet or full email exchanges for Trump's presidential campaign planning in Ohio. Yet these all influence us.

Re: Can language models serve as text-based world simulators?

#30

Post by therobots927 »

sk11001 wrote: post #17

Recently I'm getting some strange behavior from ChatGPT (with GPT-4) where it outputs a code snippet in which it declares a variable, and then on the very next line it forgets the name of the variable and refers to it as something else. Or it refers to the variable but it makes a typo in the name. If that's the behavior by one of the best models on a few lines of code, I'm not hopeful for a world simulator any time s…

I feel like it’s an open secret at this point that chatGPT is not useful for anything requiring consistent, logical thought which is a requirement for most human jobs.
Post reply on HN