Letting AI play my game – building an agentic test harness to help play-testing

I landed on something similar for my own game, though it's been pretty tricky.

I'm building a physics-based 2d game involving slingshotting around planets. The realtime nature of it has meant that it's nearly impossible for the AI to test using a browser mcp. It'll take one screenshot, then another, and in the intervening time the player shot off the map and into deep space.

Instead I gave it both a code-level api to step forward and backward the physics engine and a browser-based, `window.game` api to do it via a browser mcp console. The former helps it work out physics bugs and the latter helps it test animation and UI issues.

It's still not great. I keep occasionally getting "I tested it and it works perfectly!" as I stare at the mcp'd browser with the player stuck clipped halfway into a planet. I think, if anything, I need to lean harder into this approach: building really solid tooling for the AI to inspect every aspect of state. I would kill for a turn-based game like OP XD

I seem to remember the fatal flaw with harvester AI was that once a harvester was returning to the drop-off building, it would "claim" it, and so any other harvesters would just do a dance around the building until the the first harvester arrived. As a result, a harvester that was further away could block closer trucks if it just happened to fill sooner.

Hey this is really cool! And your game is really inventive I’d love to try it when I’m home from work.

Have you considered NOT using an LLM to test your game? Because your game is turn based and text based, could you separate rendering and logic entirely (you may have already done this by the sounds of it) and run a headless simulator that simulates thousands of games using a monte-Carlo type method? Is your game fully deterministic outside of player input?

Reason I ask is I’m making a game, it’s fully deterministic the only randomness is player input. But same inputs = same outputs from my traditional AI enemies.

With this in mind, I was able to completely separate rendering and game logic, and to tune my enemy AI (traditional AI not LLM) I can run millions of simulated games headless and generate reports of the games, and basically toggle AI parameters automatically each game until my AI is “perfect” for its archetype signature.

I can run tens to hundreds of games in parallel, and I can run a typical 5 minute game in seconds.

Then I can capture that game and recreate it and watch replays etc.

My game is also a browser game, but I built my own engine for it from scratch and no external libraries

[dead]

Thank you. You have a great suggestion. I didn't do that, but I did consider it and I think it can be very powerful. I had 2 example use cases that having an actual AI felt good for, first validating a new feature based on the spec, and second, finding unexpected bugs (like trying to enter a locked room through the back wall). It didn't do so well on the latter, but did great on the former. Having a million simulated games could probably catch those, but how would you track the reports after? Perhaps using an LLM to read the logs/reports could be a good use. Your set up sounds awesome, nice work.