I wrote a rules engine in rust along with a reinforcement learning with MCTS based system to play decks against each other. It can handle aggro decks well enough but complex combo decks like Amulet Titan are tough to get working without expert demos or reward hacking.
MTG Bench: Testing how well LLMs can play Magic
21–30 of 36 posts
Re: MTG Bench: Testing how well LLMs can play Magic
#22I know the author specifically did not use a rules engine in their simulation because of uncertainty on how it would affect it. I do still wonder if adapting something like card forge for llm use would result in engaging gameplay with an llm. https://github.com/Card-Forge/forge
MTG: Arena uses a rules engine CLIPS (a s-expr expert system based on the RETE engine), which an acquaintance wrote a course for: https://ryjo.codes/tour-of-clips.html and even a declarative chat server: https://ryjo.codes/articles/a-simple-tcp-server-written-in-g...
(defrule connection
(connection ?id)
=>
(println "User " ?id " connected")
(printout ?id "Welcome to the chatroom from CLIPS!" crlf)
(do-for-all-facts ((?f connection)) (neq ?id (nth$ 1 ?f:implied))
(printout (nth$ 1 ?f:implied) "User " ?id " connected" crlf)))
(defrule say
(connection ?id)
?f
(retract ?f ?ff)
(printout ?id "You: " ?message crlf)
(do-for-all-facts ((?f connection)) (neq ?id (nth$ 1 ?f:implied))
(printout (nth$ 1 ?f:implied)
?id ": " ?message crlf)))Re: MTG Bench: Testing how well LLMs can play Magic
#23Sadly this benchmark removes the part of MTG that is most interesting: the opponent(s). Without opponents you simply don't have a game. You just have a rules engine - quite boring! I think I object more to the decks used in testing than the machines' decisions. I do have nit picks though: This hand is quite poor and should be mulliganned: https://app.mtgautodeck.com/public/benchmarks/4bd9955b-ebe1-... . The poor runo…
Gotta walk before you can run.
Re: MTG Bench: Testing how well LLMs can play Magic
#24You don't explain how scoring works, maybe it's obvious to MTG players? If you're using gpt 5.5, is there a possibility that it is biased in favour of models that think the way it does?
Re: MTG Bench: Testing how well LLMs can play Magic
#25This is a really interesting benchmark and also timely given a lot of existing benchmarks don't do a good job. The mechanics and edge cases seem notoriously difficult to parse also even for perhaps human players. Have you been also plugging these into newer reasoning models to see how providing them with thinking time improves their win rate against the baseline?
Re: MTG Bench: Testing how well LLMs can play Magic
#26Earlier quoted context omitted.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
I was wondering how complicated it could really be, and it turns out that some people showed in 2019 that it's Turing-complete -- meaning that any conceivable computation can be simulated by a MTG game, indeed a game in which every move by every player is forced : https://arxiv.org/abs/1904.09828 IOW, it's as complicated as possible.
Someone made a video based on the paper, if you want to see the cards being used and a little more explanation: https://www.youtube.com/watch?v=pdmODVYPDLA
Re: MTG Bench: Testing how well LLMs can play Magic
#27You don't explain how scoring works, maybe it's obvious to MTG players? If you're using gpt 5.5, is there a possibility that it is biased in favour of models that think the way it does?
The scoring is just based on a simple prompt which is given the game state at the start and end of the turn and the log of tool calls and the final turn summary. The prompt asks it to evaluate the quality of the simulation from 0 to 10, and to give pass or fail for if it is legal.
It is far from ideal, but from my testing, even underpowered small LLMs that could not complete a single legal turn were reasonably good at judging if a simulation was legal. The final judging was all done by gpt-5.5 (medium) which might have given the OpenAI models an advantage, but from all the simulations I looked at, it seemed pretty fair.
This benchmark ended up be more of a test of how well an LLM can call tools without contradicting itself or backtracking. Most of the failures were not because of breaking magic rules, but because it could not sequence the tool calls correctly.
For example: https://app.mtgautodeck.com/public/benchmarks/6349dda2-4069-...
and: https://app.mtgautodeck.com/public/benchmarks/dcc18bd8-339d-...
The failure mode seems to be that some models are overly trained to start tool calls, even when the model itself knows that it should not be calling the tool. Both of those examples were not errors because the judge prompt said they were illegal. In both of those examples the model stopped the simulation itself knowing that it made a tool error.
The Opus 4.8 examples are especially weird because it will consistently make the same tool call error 2 or 3 times in a row, and it will put things like "placeholder" or "noop" for the tool call reason.
Re: MTG Bench: Testing how well LLMs can play Magic
#28This is a really interesting benchmark and also timely given a lot of existing benchmarks don't do a good job. The mechanics and edge cases seem notoriously difficult to parse also even for perhaps human players. Have you been also plugging these into newer reasoning models to see how providing them with thinking time improves their win rate against the baseline?
Since the library tools are just an MCP server, I did some testing on ChatGPT and Claude where I don't have to pay for api credits.
With maximum thinking and web search to look up magic rules, I didn't ever see it make a mistake. It is probably better at following the rules than the average magic player (but not better at making the most strategic moves).
The benchmark was mostly to find out what is the cheapest model with the lowest reasoning effort would provide a good experience for the app. The answer turned out to be that, for now, there is no cost effective way to run this app.
To provide a good experience, the simulations either need to be near instant, or you need to be able to run dozens or hundreds of simulations in parallel and do statistical analysis.
Re: MTG Bench: Testing how well LLMs can play Magic
#29I have a version of this where I have the llms play the duel decks "Elves vs Goblin" against each other using xMage as a rules engine.
Unfortunetly it gets really expensive to run even with some optimizations for the context.
I can only afford to play them with the deepseek models. They make serious blunts sometimes. This is not an easy "harness" to build and I dont have the time or disposal cash to work on it. I think a lot of work could be done on improving it still and testing better models.
It would make an amazing "arena" bench. There is plenty of more duel decks well balanced against each other.
Re: MTG Bench: Testing how well LLMs can play Magic
#30Earlier quoted context omitted.
Magic is complicated . I looked at doing something like this but the open-ended nature where one specific card will completely change the rules or require a series of followup events or modifications to the rules engine at hand is just tremendous.
or, that certain cards when play together make an infinite loop, and so cannot be played/insta-die
There is also this Matt Parker video about MTG, in which he explores a specific three-card combination that produces an ungodly amount of creature tokens.