I have been imagining this since GPT-3. I still only know of very few games making use of LLMs. There are add-ons for Skyrim, and web search points me to NVIDIA's ACE autonomous game characters framework and games such as InZOI, PUBG's ACE-powered AI teammate, Dead Meat (unreleased interrogation game), AI Dungeon, Suck Up!, Vaudeville, and more. It seems there are a few more than I would have thought, but the smaller ones seem to have some mixed reviews. It would be interesting to check out how these games resolve the reasons against LLMs in games I can think of:
Cost: LLMs fitting into local GPUs, in addition to the memory required for game textures, are too constrained for the quality, latency, and concurrency required. Connecting to the cloud would be too expensive, and even if it would work, it would indeed be the death of local offline gaming, and it would give a valid reason for a subscription to keep the servers running as opposed to a buy-once play-forever game business model.
Unreliability: You don't want to miss a major quest because the LLM did not decide to hand it out. It might be workable with some proper "agentic harness", with some hardcoded world lore, long-term memories, goals, midterm memories, some deterministic actions, and a short-term free-form loop that gets cleared every now and then. For OpenClaw instances, it would be the SOUL.md, but it would have to be read-only. Some logic might have to be hardcoded outside of LLMs, e.g., to start a new narrative branch powered by a new character memory when a keyword has been mentioned or a quest item has been given. Similar to agentic frameworks, it would have tool calls such as `offer_quest`, `give_reward`, etc.
These ideas seem to converge onto the Westworld mechanics of backstories, cornerstone memories, reveries, narratives, and loops, at least as far as I remember them from the series. Thinking about it, given enough investment, I feel like the amusement park of Westworld should already be somewhat possible to implement with the recent advances in LLMs and robotics.
In the end, I am not sure how much sense it makes for games. Maybe it would suffice to only make companion characters LLM-enhanced while NPCs stay pre-programmed in order to let the game stay deterministic. Games should simplify real life in order to increase enjoyability. Making all characters backed by LLMs might decrease enjoyability because of forgetfulness or confabulations inconsistent with the game world and actions. It would need to be heavily constrained and trained or prompted to say "I don't know about that.". It might also be questionable how much needs to be dynamic. E.g., when thinking about the NPCs in The Witcher, which are talking about random stuff, maybe you only need to pregenerate much more random stuff for them to talk about so that you don't notice the repetition, but there might be no need to have this completely dynamic and burn through GPU power for every action. The question would be what to cache and what really needs to be dynamic.