Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Letting AI play my game – building an agentic test harness to help play-testing

blog.jeffschomay.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Letting AI play my game – building an agentic test harness to help play-testing

#31

Post by jschomay »

shnippi wrote: post #9

This is sick, thanks for sharing! We've been working on very similar things for the past 2 years. We also started with a text-only representation, but sadly quickly realized that only a small subset of games work well with this. So we went down a rabbit hole and decided to do everything purely based on pixels and OS inputs. We're currently only live for mobile but happy to give you early access to nunu ai for PC if i…

Hi, thanks! That sounds really interesting. I'm curious how it compares too. What is the best way to get in touch?

Re: Letting AI play my game – building an agentic test harness to help play-testing

#32

Post by purple-leafy »

jschomay wrote: post #29
purple-leafy wrote: post #25

Hey this is really cool! And your game is really inventive I’d love to try it when I’m home from work. Have you considered NOT using an LLM to test your game? Because your game is turn based and text based, could you separate rendering and logic entirely (you may have already done this by the sounds of it) and run a headless simulator that simulates thousands of games using a monte-Carlo type method? Is your game ful…

Thank you. You have a great suggestion. I didn't do that, but I did consider it and I think it can be very powerful. I had 2 example use cases that having an actual AI felt good for, first validating a new feature based on the spec, and second, finding unexpected bugs (like trying to enter a locked room through the back wall). It didn't do so well on the latter, but did great on the former. Having a million simulated…

You’re welcome :) for you I’d recommend try get 10 games running / simulated first, and manually analyse the reports yourself to see if the report data is useful. Try and get the report data into a useful shape, and have it as either a json array or an excel. Then you can feed it into an llm to analyse.

For example for me my reports will basically be data points per AI archetype - like how often they collide with a wall, how often they perform certain actions, how often they get blocked or go idle. Straight numbers or booleans. This plus an ELO type system to rate the AI against one another so I can have an AI tier list. Then I can get an LLM to ingest the data and pick out issues / outliers etc.

My game is kinda like chess so this all makes sense for my game.

And thanks for the insights I will try a similar llm setup for manually playing my game, it’s definitely possible and it’s inspiring from your blog

Re: Letting AI play my game – building an agentic test harness to help play-testing

#33

Post by deadbabe »

purple-leafy wrote: post #25

Hey this is really cool! And your game is really inventive I’d love to try it when I’m home from work. Have you considered NOT using an LLM to test your game? Because your game is turn based and text based, could you separate rendering and logic entirely (you may have already done this by the sounds of it) and run a headless simulator that simulates thousands of games using a monte-Carlo type method? Is your game ful…

This is the way to have a very tightly balanced game. I’ve seen people come up with a lot of sophisticated graphs and curves of various params and inputs that I personally don’t understand, but they tune things to values that naturally result in the kind of outcomes players will enjoy best. It would be impossible to just tweak all these variables and their interactions just through manual play tests alone.

Re: Letting AI play my game – building an agentic test harness to help play-testing

#34

Post by jjgtvkvychnbg »

moconnor wrote: post #26

This is the future of all software; the benefits of making it accessible to agents are overwhelming.

I don’t want to play a game that was playtested by an AI (and likely built by one as well). Games have a certain aspect that requires such finesse that no machine will ever be able to solve, test, capture or design: fun.

If your game was built and playtested by AI, it is not a game worth playing.

Re: Letting AI play my game – building an agentic test harness to help play-testing

#36

Post by shnippi »

jschomay wrote: post #31
shnippi wrote: post #9

This is sick, thanks for sharing! We've been working on very similar things for the past 2 years. We also started with a text-only representation, but sadly quickly realized that only a small subset of games work well with this. So we went down a rabbit hole and decided to do everything purely based on pixels and OS inputs. We're currently only live for mobile but happy to give you early access to nunu ai for PC if i…

Hi, thanks! That sounds really interesting. I'm curious how it compares too. What is the best way to get in touch?

send me an email at jan [at] nunu [dot] ai !

Re: Letting AI play my game – building an agentic test harness to help play-testing

#37

Post by zbikowski »

I like the game, but I would suggest giving the option to disable the tilted map. I have no idea what the practical purpose for this is - I find myself tilting my head while playing!

Edit: it would also be useful to be able to see the whole dungeon at once, legibly. Maybe a larger font size or something more readable? I find myself having to write down longer words to try and fill in the gaps.

Re: Letting AI play my game – building an agentic test harness to help play-testing

#38

Post by empath75 »

ticulatedspline wrote: post #17
empath75 wrote: post #12

Earlier quoted context omitted.

I forked evennia and added it. Took me a few hours with claude. Once i had the core authorship mcp's working, claude itself created the whole world, including an initial tutorial sequence, combat, etc...

Cool, I was thinking about this very thing. Was looking at CoffeeMud and wondered if I gave it a starting room and a clean slate if it could basically just build out a whole Mud from scratch.

The answer is, yes, it will. And it will play test one, and even handles multiplayer quite nicely.

Re: Letting AI play my game – building an agentic test harness to help play-testing

#39

Post by jschomay »

zbikowski wrote: post #37

I like the game, but I would suggest giving the option to disable the tilted map. I have no idea what the practical purpose for this is - I find myself tilting my head while playing! Edit: it would also be useful to be able to see the whole dungeon at once, legibly. Maybe a larger font size or something more readable? I find myself having to write down longer words to try and fill in the gaps.

Thanks for the suggestions. I thought tilted was less rigid and matched the aesthetic of the dungeon maps at https://watabou.itch.io/one-page-dungeon which I like. But I hear you. You can press space to see the full puzzle, but it can be hard to read the words on a large level. That was one UI concern I didn't quite figure out. I'll consider your suggestions!

Re: Letting AI play my game – building an agentic test harness to help play-testing

#40

Post by SebastianSosa »

Edit: let me begin with this is fucking brilliant!!

What if the event flow has a time component?

My naive guess is make the time ticker a tool claude can control time.

Which would bifurcate the control surface into two classes: user accessible (keypress) and user inaccessible (fixtures, time).

"tick" would not work for small intervals.

Instead you would need "progress_x_time".

The concept of non-linear problems lingers in my mind.

Post reply on HN