Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning Mistral 7B on Magic the Gathering Draft

generallyintelligent.substack.com

Post reply on HN
Print view View original

91–100 of 101 posts

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#91

Post by maypop »

dharmab wrote: post #49
zoogeny wrote: post #26

I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge. One thing it did make me think about was that these models are suitable for things that don't have a natural definitive answer. That is, picking the perfect card given a set of picks is probably combinatoriall…

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

They can’t do super human performance like alpha go and the can’t think “system 2” that would be required for high stakes decisions.*

Room to grow.

*observations from the recent karpathy llm talk.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#92

Post by Tostino »

float-trip wrote: post #63
Tostino wrote: post #58

Earlier quoted context omitted.

Try doing the same thing in your dataset, but don't actually add them as "special tokens", and just let them just be multiple tokens. Adding new tokens needs a ton of data to train what the token means. Reusing existing tokens, will allow you to easily teach that a sequence of tokens now has a new meaning after fine tuning.

That's what I ended up doing (`[Author] username [Title] post title...`) > Adding new tokens needs a ton of data to train what the token means. But how much? 300M tokens is fine for a simple version of ChatML with ~4 tokens. Not for 15, at least in my case. How's this relationship scale? Just trying to offer one datapoint for what doesn't work, with the hedge that I might have just had a bug

I don't know how many tokens are required to get good results, because I simply didn't mark mine as "special_tokens" due to the issues that I had read about. I got great results, whereas others who have tried special tokens got pretty poor results. I'm sure there is a magic number, but it's just not been worth it for me to explore that area yet.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#93

Post by bart_spoon »

mtnGoat wrote: post #82

I think Yahoo fantasy sports and others in the space are doing amazing work on this idea. I wonder if an LLM is even necessary for this, it’s mostly maths. Analyze past winning decks and made decisions based on performance.

I think a non-LLM model would be good for learning to draft within a particular set, because it can simply encode each card and learn what cards do well together, comparative strength, etc. without learning the stats/rules text on the card.

For learning to draft in one environment and then applying it to a new set of cards, like done here, it would become far more difficult to do without an LLM. The variations and nuances of cards rules text would have to be encoded as features, which would be extremely cumbersome. The LLM gets you some level of understanding of that for free.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#94

Post by sdwr »

doctorpangloss wrote: post #40
dmakian wrote: post #37

Earlier quoted context omitted.

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Definitely not perfect data though, and agree that defining good in this context is hard -- a lot of the variance of "good" depends on how you play the cards either way. All good points!

> I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Hmm, but there are a lot of players with greater than a 62% lifetime win rate with very few drafts, but there may be many of those players... do you see? The win rate isn't a good filter. You chose it, you are trying to justify it, and I'm not convinced, not without the hard numbers. I'm not confu…

A lot of words to say "good players can't actually be good"

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#95

Post by Ntrails »

I don't like accuracy as a loss metric - you could end up with a similar viable draft/deck at 50% accuracy or a completely non-viable one.

Is it managing mana curves, creature count, mana fixing priority etc? I'd also love to know what the common characteristics of "missing" are. Does it undervalue a splash, or removal, or fast/slow.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#96

Post by cainxinth »

zoogeny wrote: post #26

I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge. One thing it did make me think about was that these models are suitable for things that don't have a natural definitive answer. That is, picking the perfect card given a set of picks is probably combinatoriall…

>I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge.

Surely, but we can't gloss over the fact that this was accomplished by a single person.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#97

Post by entropicdrifter »

dharmab wrote: post #80
nothrowaways wrote: post #67

Earlier quoted context omitted.

What are examples of low stakes

Generating content for tabletop gaming with my friends (especially wacky ideas, like character names themed after items on the Taco Bell menu) I had to buy some spare tools where I cared more about price than quality and it helped me choose some suitable brands As mentioned, you can tell it a bit about a person (and feed in their wishlist if they have one) and it'll help you pick something they'll probably like Findi…

> Generating content for tabletop gaming with my friends (especially wacky ideas, like character names themed after items on the Taco Bell menu)

I've been going ham with this. Pay for ChatGPT Plus for work, so GPT-4's been helping me design encounters, plan sessions, and brainstorm ideas for one-shots. It gives me a sort of bland, vanilla idea for something, I suggest a twist on it, it goes, "oh great idea, here that is with your changes:" and I iterate with it from there.

Likewise I love theming characters, plotlines, and settings after songs, bands and albums, so I'll dump in a bunch of lyrics and ask ChatGPT to help me intertwine subtle references into descriptions, names, and plot points.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#98

Post by zoogeny »

cainxinth wrote: post #96
zoogeny wrote: post #26

I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge. One thing it did make me think about was that these models are suitable for things that don't have a natural definitive answer. That is, picking the perfect card given a set of picks is probably combinatoriall…

>I like that this shows how hard even conceptually simple ideas are to achieve in fine-tuning LLMs. Even given a pretty good starting dataset, a decent starting model, etc. this appears to have been a challenge. Surely, but we can't gloss over the fact that this was accomplished by a single person.

Yes and no I think. I've seen individuals achieve things in their bedrooms that would make most corporations blush. Demoscene type stuff comes to mind as an example. Often a single person can become hyper obsessed with achieving some goal and in the absence of any interference can achieve something impressive beyond what can be achieved within a company.

Consider a PM involved in this project, feeding in requirements from a business. Instead of the "just get it done at any cost" mentality of a single person you would have KPIs and business objectives that would muddy the water.

I just mean to say that there is a gulf between what can be done by a single hacker in his basement when they have no constraints other than their imagination compared to what can be accomplished by a business. Sometimes the single-hacker achievement doesn't scale.

So, it is impressive that this is possible for a single person at all. But, from a business/operation perspective, I don't actually think that is as relevant as it may seem.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#99

Post by tigger0jk »

sciolist wrote: post #69
mdaniel wrote: post #55

Earlier quoted context omitted.

I hope I also did not accuse it of being good, but the observation I was trying to make is that -- according to the article, I have not myself confirmed the claim -- they run the card evaluation logic and gameplanning locally, not in a data center full of H100s, which I consider to be quite a feat given the free-text-y self-modifying rules of M:TG

One of the big things to note is that Sparky plays very basic decks, with few complicated cards and combos. Rules-based AI could definitely play at a basic level using a beatdown strategy, but give it some sort of control/combo deck and it would struggle.

Hearthstone allegedly had bots hit Legend rank back in 2014 playing Aggro Shaman. Those were believed to be pretty simple rules bots, so yeah +1 that approach gets you pretty far if you have decks crafted for it.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#100

Post by HanClinto »

danbrooks wrote: post #3

Super interesting that drafts can be represented with LLMs. The best performing draft AI's I've seen leverage representation learning in some form. See: https://arxiv.org/pdf/2107.04438.pdf

Unless I'm misreading something, it appears that the linked paper appears to use one-hot encoding to represent each of the cards -- not any learned embedding to represent each card -- unless I'm misunderstanding what you mean by "representation learning"?
Post reply on HN