Based on my experiences with this. Keeping context relevant is the hardest part. Your human GM will remember that funny remark from the session 3 years ago, but the LLM?
Maybe this is a solution https://www.letta.com/
LLM based agents as Dungeon Masters
11–20 of 144 posts
Re: LLM based agents as Dungeon Masters
#12Alas, I was desperately keen to read transcripts of actual gameplay and all we got was user surveys?
Was there a structured combat system as sort of this or was it a talking only kind of thing? Who knows?
Trying to get empirical evidence that an LLM can even remotely plausibly work as a DM with no social aspect and (as far as I can tell, reading the paper) no programmed system of stats, rolls, combat etc. seems impressive.
…but was it that? Or just a DnD themed chatGPT conversation? Or a text adventure with an LLM describing each “room”?
Did the LLM actually have agency to make decisions and perform tasks, or was it not an agent at all, just a conversation?
Without a transcript, all you can really say for sure is yeah, I agree; more research is needed.
Re: LLM based agents as Dungeon Masters
#13The main problem with doing this on chatgpt is that it is _relentlessly_ cheerful and it's almost impossible for it to produce any kind of interpersonal drama or threat or danger.
I tried to get it to run space opera about a diplomatic mission to a newly recontacted planet, full of Game of Thrones-style intrigue and plotting and scheming, and it was like playing with a cheerily optimistic new UN intern that completely believes in the power of compromise and diplomacy to solve all problems.
It was always like: "And so the two of you hashed out your differences over a glass of Dubranian Forblik, and peace reigned forever and ever after."
That's true of how it does role playing games and any kind of fiction at all -- it always wants to tie a neat bow on it at the end of every few paragraphs with a moral and everything.
You also have to ask it to use python to dice roll for anything you want to potentially fail on, or it will always make everything you do succeed wonderfully.
I will say this -- the advanced audio mode for D&D is amazing and it really does act out the characters and it will occasionally slip into different voices or even add sound effects, even though it's not "allowed" to. It will also sometimes copy your voice and act out what it thinks you should do instead of letting you talk, though.
Re: LLM based agents as Dungeon Masters
#14Yeah, completely automatic DMing just makes it like an open-ended computer game. On the other hand, the AI generated NPC responses are better than I could have come up with quickly, which makes me think co-DMing with an AI partner might be fun, particularly if they get better at remembering the details. That would let the human DM control the flow of the story without worrying that Thug #4 has taken 3hp of damage and is Dazzled. Of course, I most often use Foundry or Roll20 these days and they do a lot of that for you.
I wonder how well it could do for game summaries and recaps?
Re: LLM based agents as Dungeon Masters
#15I find this idea fascinating, and I think a lot of very interesting emergent behavior can be added to video games by having something like an LLM act as a Dungeon Master. Rather than having to pre-program a bunch of choices that the player is allowed to pick between, have an AI model evolve the story and just move the pieces on the story board based on what the player actually does. Story driven games these days are still very much railroaded. They are like interactive movies. The player has very limited agency. They can't really pick any choice which the designers haven't thought to implement upfront.
At the same time, I'm personally not super enthusiastic about using LLMs for a pen and paper game. Let's assume for a moment that we have free will, then the most fun aspect of PnP dungeon mastering is the fact that the DM has agency. It adds an (IMO) very fun mini-game where the players try to get away with as much shenanigans as they can, and the DMs job is to try to thwart these shenanigans in the most entertaining ways possible. The keyword is on "try" - the GM has to put entertaining obstacles in front of the players to keep the game fun, but ultimately the players should have an avenue of success. Even if they don't always succeed, they need to succeed enough and when they fail they need to feel that it was fair.
I feel that with an AI DM, this goes out the window. At least with the current crop of AI, I would never feel that I'm facing an agent with it's own free will, rather just a machine. As such, it cannot replace a human DM for the PnP purposes.
Re: LLM based agents as Dungeon Masters
#16Last year, I built a DM for myself using the OpenAI api and an elevenLabs voice generator. I asked it set my character in Baldur's Gate, so it could pull upon the huge amount of DnD source material it had been trained on.
A few takeaways:
1. An LLM based DM can give the player essentially infinite richness and description on anything they ask for.
2. There is difficulty in setting the rules for the LLM to follow that match the DnD rulebook. But this is possible to solve for. Also, I found the LLM to be too pliable as a DM. I kept getting my way, or getting my hand held through scenarios. Maybe this is a feature?
3. My conversation quickly began to approach the context window for the LLM and some RAG engineering is very necessary to keep the LLM informed about the key parts of your history.
4. Most importantly, I found that I most enjoy the human connection that I get through DnD and an LLM with a voice doesn't really satisfy that.
Re: LLM based agents as Dungeon Masters
#17Earlier quoted context omitted.
I'd love to see some human & machine co-op for DM. That'd be super helpful.
Now this is solid, like an ai enhanced Evernote, but for organization, not generation/creation.
I've been meaning to take a crack at doing this - I'd like to use Whisper to transcribe session recordings and then have an LLM build out a campaign wiki. I enjoy being a DM but note taking and organization is my least favourite part.
Re: LLM based agents as Dungeon Masters
#18The main problem with doing this on chatgpt is that it is _relentlessly_ cheerful and it's almost impossible for it to produce any kind of interpersonal drama or threat or danger. I tried to get it to run space opera about a diplomatic mission to a newly recontacted planet, full of Game of Thrones-style intrigue and plotting and scheming, and it was like playing with a cheerily optimistic new UN intern that completel…
Reminiscent of the Twilight Zone episode where a criminal ends up in the afterlife. Spoilers in link https://en.wikipedia.org/wiki/A_Nice_Place_to_Visit
Re: LLM based agents as Dungeon Masters
#19This is so interesting that people want this. This is beyond the guy saying “people don’t like making music so we made ai so it’s easier and faster to just prompt it to make music” . Like in my experience the best part of d&d is the dungeon master who knows their friends making a story and details specifically for fun and the fun of the group. And the interactions between players and the dm is where I derived the mos…
The modern thing to do would be to automate the game, watch the bots play each other, and make a reaction video for TikTok.
Re: LLM based agents as Dungeon Masters
#20Based on my experiences with this. Keeping context relevant is the hardest part. Your human GM will remember that funny remark from the session 3 years ago, but the LLM?
I'd love to see some human & machine co-op for DM. That'd be super helpful.
I've used LLM's in this capacity, and it's awesome. It quickly becomes a crutch.