Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning Mistral 7B on Magic the Gathering Draft

generallyintelligent.substack.com

Post reply on HN
Print view View original

71–80 of 101 posts

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#71

Post by algo_trader »

freediver wrote: post #15

Super interesting work. Do you have thoughts how to leverage this to create a deck builder AI that would also simulate games? The major problem here is that the search space for MTG is amazingly vast. I've seen this effort previously, pretty exciting stuff: https://www.youtube.com/watch?v=Xq4T44EvPvo

> https://www.youtube.com/watch?v=Xq4T44EvPvo

Why would he need to write a game simulator from scratch?

Surely there are OSS versions which are "good enough" ?

(disclosure: not a MtG expert)

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#72

Post by nlpfromscratch »

objektif wrote: post #66
8f2ab37a-ed6c wrote: post #57

Earlier quoted context omitted.

Search for articles showing you code for fine-tuning Llama 2, ideally including a colab notebook that you can run and modify yourself so that you have real code to work with. You can try to modify their working example to suit your own toy project as a first step.

This has not been very useful as everyone is trying to market their tool in a way.

Assuming you are using Transformers, the official notebooks are a logical place to start: https://huggingface.co/docs/transformers/notebooks

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#73

Post by pclmulqdq »

Palmik wrote: post #44
dmakian wrote: post #30

Earlier quoted context omitted.

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Otherwise, you're not measuring the best players on the service. You're training on draft choices where most choices were very good - i.e., win rate sort will show you the luckiest players, not the best ones. That will naturally show up in any v…

In ELO like match-making, you typically pair together people such that they are likely to have 50% chance to win. Therefore as the OP says, filtering down to people with high (60+%) life-time win-rate creates some sort of (interesting) bias. I would select from all games played on sufficiently high level.

They don't fully use Elo for matchmaking. There's a league system, and you get matched with players in your league. The ranks reset frequently, too.

Edit - I did the math. From the data on the MTG Elo Project, top Magic players have about a 70-75% game win percentage over an average tournament player. They have the top player at ~2300 Elo with the average being around 1500 (in matches), and have scaled the Elo system so that a 200 point gap is a 60% chance to win a best-of-three match (this is NOT the same as Chess Elo scoring).

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#74

Post by rictic »

nothrowaways wrote: post #67
dharmab wrote: post #49

Earlier quoted context omitted.

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

What are examples of low stakes

A random sampling of things GPT-4 has helped me with lately:

Where are the dates in whole foods? (A: with nuts, not fruits and veggies)

How can I steam bao without a steamer basket? (A: saucepan, 1" water, balled up aluminum foil, plate, baos, lid)

Any guess as to when this photo was taken? It looks like anywhere from the 70s to the 90s. (A: the photo paper has a logo that postdates a 2003 company merger)

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#76

Post by donpark »

dmakian wrote: post #37
doctorpangloss wrote: post #35

Earlier quoted context omitted.

Hmm, but that will filter out more than half the players on the Match Wins and Trophies based leaderboards, many of them Diamond and Mythic. So I think your choice of 62% match win rate is almost certainly disproportionately selecting for people who received very good draft choices, even if it includes some actually very good players in the data set. I mean 62% might feel like a good number, but it's arbitrary, you'd…

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Definitely not perfect data though, and agree that defining good in this context is hard -- a lot of the variance of "good" depends on how you play the cards either way. All good points!

Data selection depends the use-case. Two contrasting use-cases I see are:

- Emulation

- Advisor

In case of MTG player emulation for example, I think it makes sense to group data by some rankable criteria like winrate to train rank-specific models that can mimic players of each rank.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#77

Post by s1artibartfast »

pixl97 wrote: post #54
dharmab wrote: post #49

Earlier quoted context omitted.

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

How much additional calculation occurs in high-stakes decisions by individuals. Also what is the variability in quality of high stakes decisions in humans? I'm guessing LLM decision is rather average, but that the LLM has no easy way of spending the extra time to gather information around said high stakes decisions like a human would.

I dont think additional calculation is the difference. It makes more sense to think of individual humans as models which are highly tuned.

Just like like LLMs, some humans are better tuned than others for specific tasks, as well as in general.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#78

Post by dmakian »

qrian wrote: post #75

I've seen models learn heuristics that are harmful in real performance, and I wonder how much is accuracy directly transferrable to actually good drafting. A question, when GPT-4 contradicts in explanation, how much of them were in fact correct?

> A question, when GPT-4 contradicts in explanation, how much of them were in fact correct?

It was mostly when a card is good in a vacuum but not as good in a specific set. WOE (which this was trained on) skewed pretty aggressive, so GPT-4 was tended to overvalue strong expensive cards (compared to what good players thought at least).

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#79

Post by float-trip »

dmakian wrote: post #48
float-trip wrote: post #47

Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short (like you have with "Cut in.") Domain adaptation over subreddits/forums before finetuning may help as well.

> did you also try using weighted loss with Axolotl This is really smart, I didn't think about this! Will add it to my list of things to try, great idea! > Domain adaptation over subreddits/forums before finetuning may help as well. I was thinking about this too (along with transcribing draft youtube videos), I'd definitely be curious how much this helps.

Related comment from gwern: https://news.ycombinator.com/item?id=38438859. Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1.

Also - why qlora rather than a full finetune? Using LambdaLabs, it'd cost roughly the same as your quote. Cheaper I think if you're willing to gamble with fp8: https://github.com/mosaicml/llm-foundry/tree/main/scripts/tr.... And fewer hyperparameters to tune as well

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#80

Post by dharmab »

nothrowaways wrote: post #67
dharmab wrote: post #49

Earlier quoted context omitted.

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

What are examples of low stakes

Generating content for tabletop gaming with my friends (especially wacky ideas, like character names themed after items on the Taco Bell menu)

I had to buy some spare tools where I cared more about price than quality and it helped me choose some suitable brands

As mentioned, you can tell it a bit about a person (and feed in their wishlist if they have one) and it'll help you pick something they'll probably like

Finding something to do to spend an afternoon in a city while traveling

In general, anything where there is no objective best answer (meaning I can ask it to generate multiple possibilities and filter out the bad ideas) and where I value speed over correctness.

Post reply on HN