Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

(Unsuccessfully) Fine-tuning GPT to play "Connections"

danielcorin.com

Post reply on HN
Print view View original

31–40 of 54 posts

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#31

Post by aidos »

matsemann wrote: post #18
RheingoldRiver wrote: post #3

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.

Also, there is no reason to limit the amount of guesses. Just let me try until I figure it out. But no, they've put a limit so that it can be "sharable" in a tweet-sized text, to try to copy the viralness of Wordle. But they do it to the detriment of the gameplay, in such a way that I don't even bother playing.

The format in the show it’s lifted from (Only Connect - greatest game show ever) is that the teams have 2 minutes total to solve the “connecting wall”. They can have as many guesses as they want until they solve the first two groups - after that it’s 3 strikes and you’re out.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#32

Post by jph00 »

firebaze wrote: post #25
xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

ChatGPT4 solved today's riddle in the first try for me. Caution, spoilers ahead: https://chat.openai.com/share/0c40a0b5-ab8f-4094-a7cc-21bb94... (it even ignored some embarrassing typos ...)

It didn't solve it -- instead it simply created groups in the exact order you provided.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#34

Post by thomasahle »

binarymax wrote: post #29

You need to model how a person actually plays connections. Start with the most obvious group that has the least ambiguity, and then your problem space is smaller on category 2, then the same for category 3 and 4. So really you could fine tune 3 models - one for 16 words, one for 12, and one for 8. Then use them in succession. Also, if you come across a mistake at the end (have some negative examples in the training s…

It might even be easier to pick an arbitrary word, and ask it to find the three that matches it.

Asking GPT to just pick any group, adds a lot of extra "mental overhead".

Though of course this works best if all the groups are roughly of the same difficulty.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#36

Post by ilaksh »

I have a couple of ideas.

1. Have it do a thinking/brainstorming phase first to try to work out what the potential categories are.

2. Then ask it to scan over each word and think about what categories it could go in, in order of likelihood.

3. Ask it to do the final answer.

Format the training set in that way, as if it got everything right at each step (since you only have the right answers).

It sounds like you had 7 * 30 = around 200 examples. Maybe you can feed a batch of ten at a time and explain the game and try to get GPT-4 to generate more examples. You will have to see if they make sense.

I assume that by increasing the size of the dataset by a factor of ten, and having the LLM think through the problem using multiple steps, you will get significantly better results.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#37

Post by ehaveman »

fun. good writeup. i tried setting up a custom gpt to run a game like anagramish.com - gave it the word list to choose from, instructions on the rules, etc - but no matter what i did in the prompt it would hallucinate start words or it would incorrectly accept invalid guesses (or mark correct guesses as invalid).

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#38

Post by benpacker »

noxvilleza wrote: post #23
benpacker wrote: post #7

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

That approach works well for a game like [Codewords]( https://en.wikipedia.org/wiki/Codenames_(board_game) ) where you're trying to find a single-word common hint between many of your words (that doesn't hit any of the other words). My feeling is that it'll struggle with word-plays in OnlyConnect/Connections (like missing letters, added letters, words-within-words homophones, etc) as well as two-step references (such…

Ah wow, I'm not a frequent player so I didn't know how clever it can get!

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#40

Post by furyofantares »

noxvilleza wrote: post #23
benpacker wrote: post #7

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

That approach works well for a game like [Codewords]( https://en.wikipedia.org/wiki/Codenames_(board_game) ) where you're trying to find a single-word common hint between many of your words (that doesn't hit any of the other words). My feeling is that it'll struggle with word-plays in OnlyConnect/Connections (like missing letters, added letters, words-within-words homophones, etc) as well as two-step references (such…

Does it?

I thought it would. But I've spent a fair bit of effort both using embeddings and also using prompts to GPT4, as well as combinations of the two approaches, to try to make a good spymaster for Codenames with essentially zero success.

I wonder if something like https://wordassociations.net/en might be better for it than embeddings.

Post reply on HN