Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning Mistral 7B on Magic the Gathering Draft

generallyintelligent.substack.com

Post reply on HN
Print view View original

21–30 of 101 posts

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#21

Post by dmakian »

freediver wrote: post #15

Super interesting work. Do you have thoughts how to leverage this to create a deck builder AI that would also simulate games? The major problem here is that the search space for MTG is amazingly vast. I've seen this effort previously, pretty exciting stuff: https://www.youtube.com/watch?v=Xq4T44EvPvo

I've definitely thought about this problem and think it's in the range of 'feasible', but it would be pretty slow and expensive given how much context you need to provide a model for it to be able to reason about the game state. Worth trying though!

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#22

Post by greysphere »

It would be interesting to compare to training a NN to draft w/o the Mistral starting point (both by epoch and by $). It's not obvious to me why the LLM component would be relevant. Maybe there are enough deck lists or mock drafts on the internet to have an influence I suppose. Or maybe 'fine tune an llm' just has more infrastructure than 'create a nn'. Maybe we need a nnfiddle to make that easier.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#23

Post by apetresc »

If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons.

If so, and if he's correct in his assumption that these sets are out of the bot's training cutoff window, then surely it's purely coincidence if it ends up being a good drafter? The bot would have literally no way to know what cards work well with its previous picks, what signals have been sent and received in the draft so far, etc. Not even the best human player could take (for example, from the sample prompt) "Gadwick's First Duel -- {1}{U} (uncommon)" and figure out what works well with that (if they've never seen the card before).

It would just end up picking generically good draft cards that share a color with its previous picks. Which is already what pick-order-based heuristics have always done.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#24

Post by dmakian »

apetresc wrote: post #23

If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. If so, and if he's correct in his assumption that these sets are out of the bot's traini…

> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons.

Not quite -- there's a few ways the model learns the full card text:

* The models are trained on card trivia completions as well, where they're asked to complete the full text of the card as well as information about it (type, CMC, etc.)

* The models do still have to learn next token completion on the cards in packs, meaning they learn to predict the full text of the cards while making draft picks as well.

Net net, the bots learn the text of the new cards pretty comprehensively.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#25

Post by apetresc »

dmakian wrote: post #24
apetresc wrote: post #23

If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. If so, and if he's correct in his assumption that these sets are out of the bot's traini…

> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. Not quite -- there's a few ways the model learns the full card text: * The models are…

Ooh I see! You do that with Mistral7B, I'm guessing? But not with the small GPT-3.5 trial you did?

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#26

Post by zoogeny »

I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge.

One thing it did make me think about was that these models are suitable for things that don't have a natural definitive answer. That is, picking the perfect card given a set of picks is probably combinatorially impossible to solve. But picking a good card given a set is possible and LLMs can approach human level performance.

I think this leads to a set of problems that current LLMs may be fine-tuned to solve.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#27

Post by dmakian »

apetresc wrote: post #25
dmakian wrote: post #24

Earlier quoted context omitted.

> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text for the cards in the pack it's being passed. It doesn't look like context is being maintained between picks, presumably for context window size reasons. Not quite -- there's a few ways the model learns the full card text: * The models are…

Ooh I see! You do that with Mistral7B, I'm guessing? But not with the small GPT-3.5 trial you did?

The two larger GPT-3.5 trials also got the card trivia examples, but like a bad scientist I don't have a great control group for those

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#28

Post by apetresc »

dmakian wrote: post #27
apetresc wrote: post #25

Earlier quoted context omitted.

Ooh I see! You do that with Mistral7B, I'm guessing? But not with the small GPT-3.5 trial you did?

The two larger GPT-3.5 trials also got the card trivia examples, but like a bad scientist I don't have a great control group for those

And also, since it seems you're the author, can you also clarify if your methodology allowed for the bot to track signals outside of the color-identity-count summary statistic you pass in the prompt? Something like allowing it to notice that a card has wheeled, or that a certain synergy piece was passed a few picks ago.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#29

Post by doctorpangloss »

> With that data, you can extract “ground truth” by looking at the draft picks made by the best players on the service (sorted by win rate).

Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any validation or testing you do too.

Shouldn't this be compared not to an LLM baseline, but to a baseline where an "Elo" style score is computed for each card compared to others from the 17lands data; then, until you have two colors, suggest the best scoring card, or when you do have color(s), suggest the best scoring card within that color or a land?

I think it is possible for the LLM to have some semblance of rules knowledge, but it is more likely that it is picking up on card rarity, costs and "Big" more than anything else for unseen cards.

Your "accuracy" on the draft seems poor. I'm not sure it means what you think it means. Are you saying that when looking at the high win rate choices, where all the choices were mostly good, you happened to pick the choice that isn't the same as the player who originated the data? It actually seems harder to make a choice among all good choices.

Anyway, there is quite a bit going on here.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#30

Post by dmakian »

doctorpangloss wrote: post #29

> With that data, you can extract “ground truth” by looking at the draft picks made by the best players on the service (sorted by win rate). Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where mo…

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any validation or testing you do too.

Ahh no just unclear in the post, I'm filtering to players in 17lands with a > 62% match win rate who are drafting at a high ranking (>=diamond rank). I look at all of those players' drafts though, even the ones where they do poorly.

> Your "accuracy" on the draft seems poor. I'm not sure it means what you think it means. Are you saying that when looking at the high win rate choices, where all the choices were mostly good, you happened to pick the choice that isn't the same as the player who originated the data? It actually seems harder to make a choice among all good choices.

Accuracy here is making the same choice from a given pack as one of the good players. Obviously subjective so not a perfect metric, but a decent check on ability to emulate a high-quality drafter.

Post reply on HN