Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fine-tuning Mistral 7B on Magic the Gathering Draft

generallyintelligent.substack.com

Post reply on HN
Print view View original

81–90 of 101 posts

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#81

Post by gdown »

algo_trader wrote: post #71
freediver wrote: post #15

Super interesting work. Do you have thoughts how to leverage this to create a deck builder AI that would also simulate games? The major problem here is that the search space for MTG is amazingly vast. I've seen this effort previously, pretty exciting stuff: https://www.youtube.com/watch?v=Xq4T44EvPvo

> https://www.youtube.com/watch?v=Xq4T44EvPvo Why would he need to write a game simulator from scratch? Surely there are OSS versions which are "good enough" ? (disclosure: not a MtG expert)

Yeah, that surprised me too, given that https://github.com/magefree/mage is open source and pretty actively developed. But watching the rest of the video it looks like his implementation only needed to support <20 different cards that he picked, so the limited rule set he needed might have been easy enough to write.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#83

Post by Mengkudulangsat »

dwrodri wrote: post #2

It's not the most revolutionary change to our daily lives, but I do genuinely look forward to playing against bots that have interesting play styles for games like Magic: the Gathering. I think this is a clear case where it could drastically improve the ability for the R&D team to come up with and test new mechanics at different levels of play.

The OpenAI Dota2 experiment produced many interesting behaviours, even the pros are impressed.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#84

Post by HanClinto »

Excellent, thank you for posting this!

I was actually just looking into fine-tuning an LLM for Magic: The Gathering this week -- I've been building a small card-similarity browser using semantic embeddings of cards to find functionally or flavorfully similar cards.

I've just been using InstructorXL, but either Instructor doesn't have enough innate knowledge of the game, or else I need to work on better prompts, but so far I've tried 9 different prompts, and none of them seem to perform very well for generating embeddings:

https://github.com/HanClinto/MtgMatrix/blob/main/data/create...

So my next step was to try and download a dataset of similar cards (I have some ideas on this), and I was trying to see if I could use this to do triplet-loss training of a large embedding model or something.

Aaaaand, that's as far as I've gotten. I haven't actually figured out _how_ to hook all of that up, but your post is extremely inspirational for me. Thank you for posting this!!

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#85

Post by sdenton4 »

dmakian wrote: post #20
matsemann wrote: post #18

How is the fine tuning actually performed? They have the data of drafts, and a prompt. But what does one do with it, more concretely?

High level it's basically: 1. Generate a lot of text examples that look like this: https://gist.githubusercontent.com/davidhershey/f57d0b19563f... 2. The model is effectively trained to predict the next token based on the previous tokens in each of these examples, which has the side effect here of teaching it to make a draft pick based on the contents of a pack. Nothing too fancy, just next word prediction more or le…

Curious how different the performance would be if instead of a 'Hall of Famer' we tell the bot that it is decently-good, but will be deactivated if it can't achieve human-level performance...

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#86

Post by FanaHOVA »

mtnGoat wrote: post #82

I think Yahoo fantasy sports and others in the space are doing amazing work on this idea. I wonder if an LLM is even necessary for this, it’s mostly maths. Analyze past winning decks and made decisions based on performance.

17Lands data would definitely help, but humans don't have that data in their head when sitting at a draft table.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#87

Post by beacon294 »

reactordev wrote: post #9

I like how it identified that you haven't committed to either white or blue yet. It was aware of deck composition and not just going for the jugular. Keep tuning. It could also be Human-bias because you also played the hand. Have someone else draft against your LLM and then you play it and see if it's the same. Statistically it should match given enough games.

draft data exists, might be a fast alternative

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#88

Post by nicbou »

pixl97 wrote: post #54
dharmab wrote: post #49

Earlier quoted context omitted.

That lines up with my experience- for high-stakes decisions, they rarely give me a great answer. But for low stakes decisions, they do well at giving me a good enough answer. For example, I've been using them to help find gifts for friends and children this month. I don't need the best choice to solve the problem, just a good one.

How much additional calculation occurs in high-stakes decisions by individuals. Also what is the variability in quality of high stakes decisions in humans? I'm guessing LLM decision is rather average, but that the LLM has no easy way of spending the extra time to gather information around said high stakes decisions like a human would.

The difference is that you can reject a low-stakes answer that's invalid. You can tell that something is off, or it doesn't matter.

With high-stakes decisions, you're surrendering the decision-making power to the AI because you don't understand the output well enough to verify it.

Basically, and AI can give ideas but not advise.

Re: Fine-tuning Mistral 7B on Magic the Gathering Draft

#90

Post by bashauma »

mtnGoat wrote: post #82

I think Yahoo fantasy sports and others in the space are doing amazing work on this idea. I wonder if an LLM is even necessary for this, it’s mostly maths. Analyze past winning decks and made decisions based on performance.

Some status (mana cost, stats) of MTG cards are numbers, but the most important part, the effect of the card, is defined in English text. So, in my opinion, this seems to be an area where LLM can work well.

MTGA and MO development team do a lot of work to put card effects into if-then rule, but unfortunately their work is not visible to the players :)

Post reply on HN