Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning Mistral 7B on Magic the Gathering Draft

generallyintelligent.substack.com

Post reply on HN
Print view View original

31–40 of 101 posts

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#31

Post by apetresc »

greysphere wrote: post #22

It would be interesting to compare to training a NN to draft w/o the Mistral starting point (both by epoch and by $). It's not obvious to me why the LLM component would be relevant. Maybe there are enough deck lists or mock drafts on the internet to have an influence I suppose. Or maybe 'fine tune an llm' just has more infrastructure than 'create a nn'. Maybe we need a nnfiddle to make that easier.

Without Mistral, how would you get it to generalize to cards it hasn't seen before? I assume by "training a NN to draft without Mistral" you mean where the input layer is just a bitmapped vector of the cards in the pack, right? The killer feature of this experiment is that it works on sets the model has never seen before and has 0 training data on, using just the text of the card. I don't think you can do that without an LLM.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#32

Post by dmakian »

apetresc wrote: post #28
dmakian wrote: post #27

Earlier quoted context omitted.

The two larger GPT-3.5 trials also got the card trivia examples, but like a bad scientist I don't have a great control group for those

And also, since it seems you're the author, can you also clarify if your methodology allowed for the bot to track signals outside of the color-identity-count summary statistic you pass in the prompt? Something like allowing it to notice that a card has wheeled, or that a certain synergy piece was passed a few picks ago.

Only the statistics you see in the prompt (which are clearly limited). I have a lot of ideas about how you could improve that context (most likely letting the AI record and track notes throughout a draft), but this one was relatively simple to implement. Definitely room for improvement!

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#34

Post by dmakian »

gigel82 wrote: post #33

For some reason I thought fine tuning is not possible without specialized hardware (A100 / H100). Where can I learn more about hardware requirements for fine tuning on consumer GPUs?

There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB VRAM they're both pretty decent fine tuning chips. I think you could probably fine-tune a model up to ~13B parameters on one of them with PEFT (https://github.com/huggingface/peft)

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#35

Post by doctorpangloss »

dmakian wrote: post #30
doctorpangloss wrote: post #29

> With that data, you can extract “ground truth” by looking at the draft picks made by the best players on the service (sorted by win rate). Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where mo…

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any v…

Hmm, but that will filter out more than half the players on the Match Wins and Trophies based leaderboards, many of them Diamond and Mythic. So I think your choice of 62% match win rate is almost certainly disproportionately selecting for people who received very good draft choices, even if it includes some actually very good players in the data set.

I mean 62% might feel like a good number, but it's arbitrary, you'd have to justify how you chose it, and just eyeballing it, it is filtering out a lot of very good players with many, many more match wins.

Perhaps you can sort by Latest Rank, and filter out people with 2 or fewer trophies. Or you will have to validate with known bad draft choices in the prompt, to see what it does. Suffice it to say, I still don't think the 17Lands data represents what you think it does.

Like without a direct discussion about measuring and accounting for luck in the draft... for all I know the data is seriously flawed. It probably isn't, but it's maybe one of many, many issues to address when dealing with strategy card game AI problems.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#37

Post by dmakian »

doctorpangloss wrote: post #35
dmakian wrote: post #30

Earlier quoted context omitted.

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any v…

Hmm, but that will filter out more than half the players on the Match Wins and Trophies based leaderboards, many of them Diamond and Mythic. So I think your choice of 62% match win rate is almost certainly disproportionately selecting for people who received very good draft choices, even if it includes some actually very good players in the data set. I mean 62% might feel like a good number, but it's arbitrary, you'd…

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts!

Definitely not perfect data though, and agree that defining good in this context is hard -- a lot of the variance of "good" depends on how you play the cards either way. All good points!

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#38

Post by greysphere »

apetresc wrote: post #31
greysphere wrote: post #22

It would be interesting to compare to training a NN to draft w/o the Mistral starting point (both by epoch and by $). It's not obvious to me why the LLM component would be relevant. Maybe there are enough deck lists or mock drafts on the internet to have an influence I suppose. Or maybe 'fine tune an llm' just has more infrastructure than 'create a nn'. Maybe we need a nnfiddle to make that easier.

Without Mistral, how would you get it to generalize to cards it hasn't seen before? I assume by "training a NN to draft without Mistral" you mean where the input layer is just a bitmapped vector of the cards in the pack, right? The killer feature of this experiment is that it works on sets the model has never seen before and has 0 training data on, using just the text of the card. I don't think you can do that withou…

That's a good point. It looks like the article hints at some success on that front. It'd be interesting to see what that means quantitatively. Interesting that this delta could even be used as a measure of the llm's value.

I'd be curious about the difference in success w/ drafts on a new 2/2 bear with a different name, and cards with a new keyword 'fizzbangitude 7' as well.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#39

Post by filterfiber »

greysphere wrote: post #22

It would be interesting to compare to training a NN to draft w/o the Mistral starting point (both by epoch and by $). It's not obvious to me why the LLM component would be relevant. Maybe there are enough deck lists or mock drafts on the internet to have an influence I suppose. Or maybe 'fine tune an llm' just has more infrastructure than 'create a nn'. Maybe we need a nnfiddle to make that easier.

The benefit of the LLMs is that the checkpoint already "understands" a lot by default. Finetuning is relatively cheap and makes many tasks such as this one perform decently well simply by shoving some data into it.

The base checkpoint takes a lot of compute to make, but that's what holds most of it's "knowledge" so to speak.

Making a NN from scratch means you'll have to somehow map the cards into inputs. I have limited knowledge of how MTG works, but most TGG have text descriptions and complex effects. Mapping text to logic is what LLMs are really good at, otherwise you're starting from scratch and will also need a relatively large amount of compute before it starts displaying any type of decent behaviour.

It's also easy for most software devs to do this - finetuning mostly consists of collecting text and feeding it into a finetuning script. You don't need to know linear algebra, what a "convolution" is, etc. to do finetuning.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#40

Post by doctorpangloss »

dmakian wrote: post #37
doctorpangloss wrote: post #35

Earlier quoted context omitted.

Hmm, but that will filter out more than half the players on the Match Wins and Trophies based leaderboards, many of them Diamond and Mythic. So I think your choice of 62% match win rate is almost certainly disproportionately selecting for people who received very good draft choices, even if it includes some actually very good players in the data set. I mean 62% might feel like a good number, but it's arbitrary, you'd…

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Definitely not perfect data though, and agree that defining good in this context is hard -- a lot of the variance of "good" depends on how you play the cards either way. All good points!

> I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts!

Hmm, but there are a lot of players with greater than a 62% lifetime win rate with very few drafts, but there may be many of those players... do you see? The win rate isn't a good filter. You chose it, you are trying to justify it, and I'm not convinced, not without the hard numbers.

I'm not confused about what filter you chose. I just think it's a bad filter, and you haven't thought very deeply about how it affects the data, which includes presumably your test and validation data - however you're choosing to test and validate, apparently by hand, by some eyeballed examples.

Anyway I think you have to compare with a non-LLM, non-random baseline to have any sense if this stuff is working at all. I could be dead wrong. I would maybe compare with a community draft picker.

Post reply on HN