Can language models serve as text-based world simulators?

Exactly. I've been working on something like this for a while using finite state machines to control the prompts. The biggest struggle I've had is creating memory. It's one thing to save a list of events and feed it into the prompts, but there are always issues interpreting it and making sure the memories are detailed enough but not an entire page.

For example, you "conquered the dungeon in level 1". If this gets saved as "conquered the dungeon" the next time you get to a new dungeon, it may think you already beat it and won't generate NPC monsters, that kind of thing.

I tried for a few months getting ChatGPT4 to work with a MUD. In my experience it's not very good at that particular task.

One problem I ran into was its ability to logically connect rooms. In a MUD you navigate by going north, south, east, west, up, or down. Not every room lets you go any direction. And usually, if you go east, in your new room, you can go west. Rarely a level creator will make this not be true. ChatGPT4 was pretty bad at it though.

Another problem was descriptions. It might mention a mountain in the distance once. But then never again. So this giant landmark was described in a single room.

It was also difficult to get it to create a fair quantity of secrets in logical places. Lots of times it would just chain together multiple secrets in a single place. If you have more than one, you want to spread it around

And finally, room layout. It tended to not be very good at this. Lots of linear layouts. It didn't have an eye towards when details should be complex rooms and when it can just be a line in a description.

So it could do it, but it created levels that weren't very fun or particularly creative, even when it came to room descriptions.

Recently I'm getting some strange behavior from ChatGPT (with GPT-4) where it outputs a code snippet in which it declares a variable, and then on the very next line it forgets the name of the variable and refers to it as something else. Or it refers to the variable but it makes a typo in the name.

If that's the behavior by one of the best models on a few lines of code, I'm not hopeful for a world simulator any time soon unless we see another big leap in model quality.

I am also working on something similar. I have written a grammar system for controlling the llm that is not context free.

One of the main challenges is to have a database of what is reality, id -> object, and then have the llm return ids for objects referenced by the user. I am doing a system where something that has no id has not been created yet.

I use story templates with labeled segments to create characters, items, location as well as events. Example: "Once upon a time there was a cowardly knight named Billy Bonkers. The named qualities become the data structure associated with the id of the new character. It can be hilarious to prompt the llm to do nerdy space humor!

I seems to be very possible to create a "narrator driven" game happening in a "real" world.

I plan on using a multimodal llm and all the stuff that is created will have an image. That combined with id-less objects existing but not having a detailed description will make qualities and objects visible in the created images also part of the world.