They should randomize games of judge tower and see who wins:
MTG Bench: Testing how well LLMs can play Magic
11–20 of 36 posts
Re: MTG Bench: Testing how well LLMs can play Magic
#12Looking forward to this metric being Goodhart lawed. Like how the strawberry example was overtrained for, or how the pelican on a bike started being used in official release posts.
Magic is complicated. I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
Re: MTG Bench: Testing how well LLMs can play Magic
#13Benchmarks like this are onto something. Next frontier of llm benchmarking
Re: MTG Bench: Testing how well LLMs can play Magic
#14Looking forward to this metric being Goodhart lawed. Like how the strawberry example was overtrained for, or how the pelican on a bike started being used in official release posts.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
or, that certain cards when play together make an infinite loop, and so cannot be played/insta-die
Re: MTG Bench: Testing how well LLMs can play Magic
#15To clarify, the more accurate description would be "Testing how well LLMs can follow the rules of Magic", right? There is no actual evaluation of how "well" they are playing?
Re: MTG Bench: Testing how well LLMs can play Magic
#16I think running them against each other with a rules engine would be more interesting. Count up illegal moves and wins/unfinished games. I think llm grading is too unreliable.
Re: MTG Bench: Testing how well LLMs can play Magic
#17I know the author specifically did not use a rules engine in their simulation because of uncertainty on how it would affect it. I do still wonder if adapting something like card forge for llm use would result in engaging gameplay with an llm. https://github.com/Card-Forge/forge
> because of uncertainty on how it would affect it.
Have the LLM submit a proposed move and either advance the game state or reply "permission denied, try again". Probably also log the number of times it happens since attempted violations seems like a valuable signal as well.
Re: MTG Bench: Testing how well LLMs can play Magic
#18I know the author specifically did not use a rules engine in their simulation because of uncertainty on how it would affect it. I do still wonder if adapting something like card forge for llm use would result in engaging gameplay with an llm. https://github.com/Card-Forge/forge
I actually considered using card forge when I started this. I mostly didn't end up using it because of how much more work it would have been. But also with a rules engine, you have to manually go though every step, and pass priority after every action. I think it makes more sense to let an LLM play magic like a person would. On early turns it is acceptable to say "I play a land and pass" without going through every p…
Those things sound less like general problems with rules engines and more like deficiencies of card forge IMO.
Re: MTG Bench: Testing how well LLMs can play Magic
#19Earlier quoted context omitted.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
or, that certain cards when play together make an infinite loop, and so cannot be played/insta-die
You misspelled insta-win. Infinite turn combos are the best.
Re: MTG Bench: Testing how well LLMs can play Magic
#20Looking forward to this metric being Goodhart lawed. Like how the strawberry example was overtrained for, or how the pelican on a bike started being used in official release posts.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
I was wondering how complicated it could really be, and it turns out that some people showed in 2019 that it's Turing-complete -- meaning that any conceivable computation can be simulated by a MTG game, indeed a game in which every move by every player is forced: https://arxiv.org/abs/1904.09828
IOW, it's as complicated as possible.