Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

(Unsuccessfully) Fine-tuning GPT to play "Connections"

danielcorin.com

Post reply on HN
Print view View original

41–50 of 54 posts

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#41

Post by nsagent »

xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

I've certainly thought about testing LLMs on Connections and I'm glad someone has. It might be possible to increase their performance, but LLMs as-is are not suited for the task.

The problem is that Connections is ultimately a search problem that requires more than simply grouping similar words. There are lots of combinations to assess. I bet if you enumerate, score, then rank all possible groupings, an LLM would perform much better.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#42

Post by nigamanth »

jw1224 wrote: post #17
xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

> This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well I would speculate it’s struggling because of the linear nature of its output, and the red-herring words which crossover between categories. Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to…

> Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to a certain combination early on will lead to a mistake later.

Aren't there already models that CAN look ahead? Or are there none?

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#43

Post by jkrems »

firebaze wrote: post #25
xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

ChatGPT4 solved today's riddle in the first try for me. Caution, spoilers ahead: https://chat.openai.com/share/0c40a0b5-ab8f-4094-a7cc-21bb94... (it even ignored some embarrassing typos ...)

Apart from the "it just explained the already ordered groups in the question" problem, it didn't even explain one of the groups correctly. "Something about coat(ing) and food" is not the correct explanation, it's missing a lateral logic step there to go from food-related to a separate meaning.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#44

Post by SamBam »

thomasahle wrote: post #34
binarymax wrote: post #29

You need to model how a person actually plays connections. Start with the most obvious group that has the least ambiguity, and then your problem space is smaller on category 2, then the same for category 3 and 4. So really you could fine tune 3 models - one for 16 words, one for 12, and one for 8. Then use them in succession. Also, if you come across a mistake at the end (have some negative examples in the training s…

It might even be easier to pick an arbitrary word, and ask it to find the three that matches it. Asking GPT to just pick any group, adds a lot of extra "mental overhead". Though of course this works best if all the groups are roughly of the same difficulty.

Connections is deliberately written so that any one word might belong to multiple groups. For example, the word "Bass" might be surrounded by "Guitar," "Drums," and "Microphone," but actually belongs to a category of "Fish," while "Guitar" might belong to the category "Air ___," and "Microphone" might belong to "Something that can be dropped."

Just making up that example, but it's very common that multiple words will all appear to be one group, and actually each one belongs to a different group.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#45

Post by SamBam »

ilaksh wrote: post #36

I have a couple of ideas. 1. Have it do a thinking/brainstorming phase first to try to work out what the potential categories are. 2. Then ask it to scan over each word and think about what categories it could go in, in order of likelihood. 3. Ask it to do the final answer. Format the training set in that way, as if it got everything right at each step (since you only have the right answers). It sounds like you had 7…

I spent about 30 minutes with GPT4 and tried lots of variations of pre-processing. I had it first list large numbers of possible categories, then try to consider one category with four words, then double-check that category and look at the remaining words, then go ahead with the next....

No matter how I instructed it to think, it frequently could not work out the very first category.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#46

Post by SamBam »

matsemann wrote: post #26
krainboltgreene wrote: post #22

Earlier quoted context omitted.

The whole point of the game is to do it within a bounded set of moves.

No, it scores you based on the number of moves used. No need for an upper bound, could've let me use 20 guesses if that's what it takes (non native speaker). But that wouldn't fit their copy&paste result formatting..

Too many people would take a scatter-shot approach to solving, even with a score that keeps track of guesses, and then those same people would get bored and disillusioned with the game because it would be boring.

The limited number of attempts is precisely what makes it a game.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#47

Post by alberto_balsam »

matsemann wrote: post #26
krainboltgreene wrote: post #22

Earlier quoted context omitted.

The whole point of the game is to do it within a bounded set of moves.

No, it scores you based on the number of moves used. No need for an upper bound, could've let me use 20 guesses if that's what it takes (non native speaker). But that wouldn't fit their copy&paste result formatting..

There is a UI for the game without the upper bound at connections.swellgarfo.com. Personally that annoys me as my friends are prone to sharing massive walls of incorrect guesses when they do badly on that site, but it sounds like it would be a good fit for you.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#48

Post by noxvilleza »

furyofantares wrote: post #40
noxvilleza wrote: post #23

Earlier quoted context omitted.

That approach works well for a game like [Codewords]( https://en.wikipedia.org/wiki/Codenames_(board_game) ) where you're trying to find a single-word common hint between many of your words (that doesn't hit any of the other words). My feeling is that it'll struggle with word-plays in OnlyConnect/Connections (like missing letters, added letters, words-within-words homophones, etc) as well as two-step references (such…

Does it? I thought it would. But I've spent a fair bit of effort both using embeddings and also using prompts to GPT4, as well as combinations of the two approaches, to try to make a good spymaster for Codenames with essentially zero success. I wonder if something like https://wordassociations.net/en might be better for it than embeddings.

I was playing a bit with embeddings in 2021. I'd played codenames online with friends in lockdown and we often had interesting boards we'd talk about, so when I saw papers like this (https://arxiv.org/abs/2105.05885) I looked into the topic. I found the suggested clues were very good, and there were some 'clue scoring' functions which correlated with the actual best spymasters. Wasn't scientifically rigorous as OPs post, but I would say it was good.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#49

Post by pomatic »

daemonologist wrote: post #15

I suspect the inability of the model to "plan ahead" is a significant contributor to its poor performance relative to a human. Being able to check a grouping to be sure it includes at least four words _and_ to check that it doesn't conflict with the other three groupings is a major advantage - it's pretty common that these puzzles include partial or incompatible red herring groups. If this is the case, performance mi…

Would step-wise instruction help with the look ahead issue. Something like: 1. Here are 16 words. Find 4 that have something in common, and list the remaining 12 words. 2. Take the remaining 12 words from the previous answer and find 4 words that have something in common, and list the remaining 8 words. etc. etc.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#50

Post by coolness »

jw1224 wrote: post #17
xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

> This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well I would speculate it’s struggling because of the linear nature of its output, and the red-herring words which crossover between categories. Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to…

> I expect if you asked it to correct its output in a followup message, it could do so without much difficulty.

I had a similar idea to the author and tried this many times, albeit with the free version of ChatGPT. After getting wrong results, I prompted it to correct them, even telling the model explicitly that a category is wrong or doesn't make sense. Nothing I did made a difference.

My two cents on why this doesn't work has to do with the fact that the answer should contain a discrete set of words given in the prompt, and importantly, they should not be duplicated. I suspect that these currents models are not very good at following the instruction "the token should appear in the answer exactly once"

Post reply on HN