Can language models serve as text-based world simulators?
1–10 of 68 posts
Re: Can language models serve as text-based world simulators?
#2Nethack FTW :)
Re: Can language models serve as text-based world simulators?
#3Nethack FTW :)
Seriously, ha, time is a flat circle.
Re: Can language models serve as text-based world simulators?
#4I made a D&D game for my kids using llama. It went surprisingly well. Lots of potential here.
https://blog.katarismo.com/2023-05-26-i-m-a-dad-i-replaced-m...
Re: Can language models serve as text-based world simulators?
#5This is neat. In theory, you could hook llama.cpp into into a GOAL-based planner (https://www.gamedevs.org/uploads/three-states-plan-ai-of-fea...) and have much better default bots navigating your nav mesh. Even better, if you record player actions as GOAL actions within the nav mesh, you can use that to fine tune the model. Or even feed it back in realtime so they learn the modus operandi of the player.
Re: Can language models serve as text-based world simulators?
#6Maybe they can, but they’re trained on conversation and stories (among other things) rather than simulations. These are more general than the log of a simulation run in how they use time. Stories can be chronological, but they can also fill in the details in any order using things like flashbacks. Or they can get really complicated with time travel stories.
So it seems like to understand a story well, an LLM would need a more sophisticated notion of time, where there is a history and events are slotted in as they’re learned from the narrative, and sometimes the whole history gets reinterpreted based on new information. (Plot twist!)
It would be fascinating if some mechanistic interpretability researcher figured out how they actually work with story time. Are there the rudiments of that kind of understanding yet?
Re: Can language models serve as text-based world simulators?
#7TL;DR: unfortunately the conclusion is, no they can't
> the best recorded performance is 59.9% on accurately simulating state transitions [...] after 10 steps, average simulation accuracy would reduce to less than 1%. Our results indicate that LLMs are not yet able to reliably act as text world simulators
Re: Can language models serve as text-based world simulators?
#8I think there is a MUD effort called LlamaTale which allows creation of this as telepresence. I’m trying to put a MUD together to try this out.
Re: Can language models serve as text-based world simulators?
#9Would be interesting to see a version of this but incorporating tool usage features of the latest models.
Re: Can language models serve as text-based world simulators?
#10Not by themselves... but with a good program built around it, managing the actual state, and very careful prompting, yes. I've been thinking about this for a while, always have the desire to make a game.