They should randomize games of judge tower and see who wins: https://mtg.fandom.com/wiki/Judge_Tower
Heads up, most of the community migrated off Fandom a little while ago. https://mtg.wiki/page/Judge_Tower
MTG Bench: Testing how well LLMs can play Magic
31–36 of 36 posts
Re: MTG Bench: Testing how well LLMs can play Magic
#32A really interesting benchmark where the llms play multiplayer decks against each other using xMage as a rules engine,in this case, a $HORIZON token to the moon(Sideways).
1. Sideways walking (100M Horizontal)
2. Sideways Pinching (Crab division only)
3. Sideways Bleating (Goat division)
4. Sideways Rattling (Skeleton division)
5. Sideways Hay Toss (Mixed division)
6. Sideways Swimming (Tide pool division)
7. Sideways Knitting (GrandMittens Invitational)
8. Sideways Stay (Meditation division)
OLYMPICS RECORDS. 1.14.2 Seconds,Holder: Pinchy 2.(60s)120 pinches,Holder:Pinchy 3.(db)110db,Holder: EIDOLONX 4.(rhythm)9.7/10,Holder:Skeletorus 5.12.3m,Holder:Satochi Goat 6.(50m)32.1 sec,Holder: Pinchy 7.(1hr)100m,Holder: GrandMittens 8.(6hours),Holder: Satochi Goat Economic boost: $CRAB up 0.0001% (Sideways as Always.) Providing them with medal count will improve their win rate against the baseline $HORIZON.
Re: MTG Bench: Testing how well LLMs can play Magic
#33I know the author specifically did not use a rules engine in their simulation because of uncertainty on how it would affect it. I do still wonder if adapting something like card forge for llm use would result in engaging gameplay with an llm. https://github.com/Card-Forge/forge
MTG: Arena uses a rules engine CLIPS (a s-expr expert system based on the RETE engine), which an acquaintance wrote a course for: https://ryjo.codes/tour-of-clips.html and even a declarative chat server: https://ryjo.codes/articles/a-simple-tcp-server-written-in-g... (defrule connection (connection ?id) => (println "User " ?id " connected") (printout ?id "Welcome to the chatroom from CLIPS!" crlf) (do-for-all-facts (…
I was about to ask why someone would reach for CLIPS over implementing their own rules engine in the language of the rest of the application (I did this once).
It’s answered on the same site. https://ryjo.codes/articles/forgoing-implicity-using-abstrac...
Thank you for sharing! A lot of good stuff here!
Re: MTG Bench: Testing how well LLMs can play Magic
#34Looking forward to this metric being Goodhart lawed. Like how the strawberry example was overtrained for, or how the pelican on a bike started being used in official release posts.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
[deleted]
Re: MTG Bench: Testing how well LLMs can play Magic
#35This is a really interesting benchmark and also timely given a lot of existing benchmarks don't do a good job. The mechanics and edge cases seem notoriously difficult to parse also even for perhaps human players. Have you been also plugging these into newer reasoning models to see how providing them with thinking time improves their win rate against the baseline?
Since the library tools are just an MCP server, I did some testing on ChatGPT and Claude where I don't have to pay for api credits. With maximum thinking and web search to look up magic rules, I didn't ever see it make a mistake. It is probably better at following the rules than the average magic player (but not better at making the most strategic moves). The benchmark was mostly to find out what is the cheapest mode…
Ahh I see, thanks for sharing more about how you experimented with this.
Re: MTG Bench: Testing how well LLMs can play Magic
#36Earlier quoted context omitted.
or, that certain cards when play together make an infinite loop, and so cannot be played/insta-die
You misspelled insta-win. Infinite turn combos are the best.
only if there's a player choice in the loop. If there's a mandatory infinite loop the game ends in a draw.