Viewing profile — dmakian

> A question, when GPT-4 contradicts in explanation, how much of them were in fact correct? It was mostly when a card is good in a vacuum but not as good in a specific set. WOE (wh…

The longest running fine tuning job took about 8 hours, so ~$5. I think if you add up all of the learning and testing I did, probably closer to ~$50 total

> did you also try using weighted loss with Axolotl This is really smart, I didn't think about this! Will add it to my list of things to try, great idea! > Domain adaptation over s…

Still not clear maybe, I'm selecting players with a 62% lifetime win rate so mostly players who have been good over a larger number of drafts! Definitely not perfect data though, a…

There is not a lot of great content out there making this clear, but basically all that matters for basic fine tuning is how much VRAM you have -- since the 3090 / 4090 have 24GB V…

Only the statistics you see in the prompt (which are clearly limited). I have a lot of ideas about how you could improve that context (most likely letting the AI record and track n…

> Do you mean that you are looking at the draft picks from https://www.17lands.com/leaderboard and then sorting by Win Rate? Didn't you mean to choose Match Wins or Trophies? Other…

The two larger GPT-3.5 trials also got the card trivia examples, but like a bad scientist I don't have a great control group for those

> If I'm reading the author's writeup correctly, the prompt he's giving the agent at each pick contains only the names of the cards in its pool so far, and only gives the full text…

I've definitely thought about this problem and think it's in the range of 'feasible', but it would be pretty slow and expensive given how much context you need to provide a model f…

High level it's basically: 1. Generate a lot of text examples that look like this: https://gist.githubusercontent.com/davidhershey/f57d0b19563f... 2. The model is effectively train…

> I was under the assumption that finetuneing LLMs was useful only when you need to change the model's tone (speak like a pirate, voldemort etc). A lot of why I tried this out was …

Would love to compare notes, drop me a email at dshersh at umich dot edu if you'd be interested!

I hadn't seen this, this is awesome! You'd think given the volume of data available that this type of method would outperform an LLM, cool results. Still some fun things about LLM …