(Unsuccessfully) Fine-tuning GPT to play "Connections" • Hacker News Board

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#51

Post by coolness » Wed, Jan 17, 2024, 9:46 AM UTC

charcircuit wrote: post #4

It's a shame you can't just see the probability distributions for the 16 words and choose them yourself that way you never hallucinate a word and the groups are always 4 words long.

This is a great idea actually, this way you could also enforce that all the words appear _exactly once_ (by rejecting during sampling words that were already in the answer) which seemed to be a significant issue for me when I tried this.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#52

Post by Yenrabbit » Wed, Jan 17, 2024, 6:33 PM UTC

benpacker wrote: post #7

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

I tried with clustering similar embeddings but it did extremely poorly (~0%) since the groupings are often deceiving with words in a group only having one small way in which they're connected and lots of spurious fake groups to throw you off. Maybe looking for groups with high similarity on only a sibset of embedding dimensions might help, but I didn't have much time to play either :) A notebook to get you going if you do want to play: https://colab.research.google.com/drive/1KJeSB9Q5XzSeT9ONUJ_...

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#53

Post by Yenrabbit » Wed, Jan 17, 2024, 6:39 PM UTC

FWIW I was able to get about 20% accuracy (perfect 4/4 groups) with ~50% of groups correct on average and most mistakes being groups with 3/4 right (so at least on the right track) with my first attempt at 0-shot prompting. The prompt goes something like this: ``` You are playing [game info] with [word list] Follow these steps: - consider possible groupings as initial brainstorming (>4 groups) - propose a first hypothesis based on the likely-looking groups - reflect on whether that grouping works - revise if needed then submit the final predictions ``` Having it start with word one of group one as the first output token seems unlikely to work from my intuition about what these models can do. Heck, I can't solve it that way! Burning some tokens on exploration and hypothesis building leaves it with the easier task of choosing plausible groups from the proposed options. System 2 thinking vs system 1 perhaps.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#54

Post by benpacker » Sun, Jan 21, 2024, 2:00 AM UTC

Yenrabbit wrote: post #52

benpacker wrote: post #7

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

I tried with clustering similar embeddings but it did extremely poorly (~0%) since the groupings are often deceiving with words in a group only having one small way in which they're connected and lots of spurious fake groups to throw you off. Maybe looking for groups with high similarity on only a sibset of embedding dimensions might help, but I didn't have much time to play either :) A notebook to get you going if y…

I definitely think trying to find similarity on a variable subset of dimensions is required. Fingers crossed I get the time to try soon