I wonder if you could use a smaller model or get better results if you treated each card as a token, gave the state of the draft as an input and the predicted token would be the card to pick. You woukd have to train from scratch with a custom tokenizer.
Fine-tuning Mistral 7B on Magic the Gathering Draft
41–50 of 101 posts
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#42In case you didn't see it, https://news.ycombinator.com/item?id=38525978 (I hacked Magic the Gathering: Arena for a 100% win rate) may interest this audience if for no other reason that the investigator discovered that Sparky, the pseudo-AI in MTGA, doesn't appear to be as stupid complicated as one may have suspected from the outside
Sparky is the Arena AI, but no one ever accused it of being a good Arena AI - it is very much only there for the new player experience of playing against a dumb computer when you're first exposed to the game and don't know the rules, or for the computer equivalent of "playing against a goldfish" a deck you made to see how it draws or combos. It's not a Chess CPU.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#43If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. If so, and if he's correct in his assumption that these sets are out of the bot's traini…
> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. Not quite -- there's a few ways the model learns the full card text: * The models are…
Haha, I don't know anything about AI training but that's a really cute trick.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#44> With that data, you can extract “ground truth” by looking at the draft picks made by the best players on the service (sorted by win rate). Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where mo…
> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any v…
In ELO like match-making, you typically pair together people such that they are likely to have 50% chance to win. Therefore as the OP says, filtering down to people with high (60+%) life-time win-rate creates some sort of (interesting) bias.
I would select from all games played on sufficiently high level.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#45For some reason I thought fine tuning is not possible without specialized hardware (A100 / H100). Where can I learn more about hardware requirements for fine tuning on consumer GPUs?
Definitely possible on even older off-the-shelf hardware. I use 24GB 4090s for 13b-sized models and have even used 12GB Titans for 7b models, admittedly at much slower rates.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#46Thanks for sharing this, I found it helpful as an addition to my homebrew curriculum for learning how to fine-tune open source LLMs.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#47Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short (like you have with "Cut in.") Domain adaptation over subreddits/forums before finetuning may help as well.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#48Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short (like you have with "Cut in.") Domain adaptation over subreddits/forums before finetuning may help as well.
> did you also try using weighted loss with Axolotl
This is really smart, I didn't think about this! Will add it to my list of things to try, great idea!
> Domain adaptation over subreddits/forums before finetuning may help as well.
I was thinking about this too (along with transcribing draft youtube videos), I'd definitely be curious how much this helps.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#49I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge. One thing it did make me think about was that these models are suitable for things that don't have a natural definitive answer. That is, picking the perfect card given a set of picks is probably combinatoriall…
That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.
Re: Fine-tuning Mistral 7B on Magic the Gathering Draft
#50I wonder if you could use a smaller model or get better results if you treated each card as a token, gave the state of the draft as an input and the predicted token would be the card to pick. You woukd have to train from scratch with a custom tokenizer.
I tried adding special tokens for a reddit-style dataset once. The format was: `usernametitle here...`
The resulting model was so much worse than just formatting everything plaintext. This was with MPT-30B, 15 special tokens, 300M training tokens, and a full finetune.
I may have made a mistake, but I haven't seen any open source finetunes successfully add a large number of tokens yet either.