Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

(Unsuccessfully) Fine-tuning GPT to play "Connections"

danielcorin.com

Post reply on HN
Print view View original

21–30 of 54 posts

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#21

Post by mtlmtlmtlmtl »

xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

Not sure how Alpha Zero is relevant to whether a transformer can play connections. Alpha zero is not a transformer and chess is not connections.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#22

Post by krainboltgreene »

matsemann wrote: post #18
RheingoldRiver wrote: post #3

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.

Also, there is no reason to limit the amount of guesses. Just let me try until I figure it out. But no, they've put a limit so that it can be "sharable" in a tweet-sized text, to try to copy the viralness of Wordle. But they do it to the detriment of the gameplay, in such a way that I don't even bother playing.

The whole point of the game is to do it within a bounded set of moves.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#23

Post by noxvilleza »

benpacker wrote: post #7

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

That approach works well for a game like [Codewords](https://en.wikipedia.org/wiki/Codenames_(board_game)) where you're trying to find a single-word common hint between many of your words (that doesn't hit any of the other words).

My feeling is that it'll struggle with word-plays in OnlyConnect/Connections (like missing letters, added letters, words-within-words homophones, etc) as well as two-step references (such as {Venice, Dream, Night, Nothing} => "last words of Shakespeare plays"}).

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#24

Post by basil-rash »

matsemann wrote: post #19
epiccoleman wrote: post #2

This is pretty interesting. Intuitively, Connections is the kind of thing I would expect GPT to not be good at, because almost every day there's something that feels kind of "out of left field" in the categories. In my experience LLMs are good at regurgitating the "standard" take on a topic, or "best practices", but lack the creativity and out-of-the-box thinking that makes Connections fun. On the other hand, it feel…

The "whole point" of embeddings is that words have a vector that represents how well that word fits into a certain categories, so words belonging together is close in that vector space. So in that sense it almost feels like this should be solvable using something simpler than a full LLM. To "just" get the embeddings of the words, and then find the groups of 4 that minimizes the total distances within the groups.

The problem is Connections is designed to use a tons of alternate definitions and other vaguities that aren’t well modeled in typical embeddings. Today’s for instance (spoilers!!) has Coat, Green, Pod, and Soup as being linked for them matching “Pea ___”. No embedding would relate them at all, unless that suffix is known a priori.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#25

Post by firebaze »

xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

ChatGPT4 solved today's riddle in the first try for me. Caution, spoilers ahead: https://chat.openai.com/share/0c40a0b5-ab8f-4094-a7cc-21bb94...

(it even ignored some embarrassing typos ...)

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#26

Post by matsemann »

krainboltgreene wrote: post #22
matsemann wrote: post #18

Earlier quoted context omitted.

Also, there is no reason to limit the amount of guesses. Just let me try until I figure it out. But no, they've put a limit so that it can be "sharable" in a tweet-sized text, to try to copy the viralness of Wordle. But they do it to the detriment of the gameplay, in such a way that I don't even bother playing.

The whole point of the game is to do it within a bounded set of moves.

No, it scores you based on the number of moves used. No need for an upper bound, could've let me use 20 guesses if that's what it takes (non native speaker). But that wouldn't fit their copy&paste result formatting..

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#27

Post by n2d4 »

matsemann wrote: post #18
RheingoldRiver wrote: post #3

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.

Also, there is no reason to limit the amount of guesses. Just let me try until I figure it out. But no, they've put a limit so that it can be "sharable" in a tweet-sized text, to try to copy the viralness of Wordle. But they do it to the detriment of the gameplay, in such a way that I don't even bother playing.

Disagreed, the limit is what gives the game a constraint and makes it interesting IMHO. I like to have something that makes me fail because I care less about optimizing a score, more about beating it in the first place. Different people play games differently, etc.

I also don't see how it makes it "sharable". Wouldn't it be more sharable if they let everyone win and just give them a score?

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#28

Post by mbb70 »

firebaze wrote: post #25
xnorswap wrote: post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

ChatGPT4 solved today's riddle in the first try for me. Caution, spoilers ahead: https://chat.openai.com/share/0c40a0b5-ab8f-4094-a7cc-21bb94... (it even ignored some embarrassing typos ...)

Doesn't this list the words in the order that they are grouped? The article states that randomizing the words completely eliminates any successful results

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#29

Post by binarymax »

You need to model how a person actually plays connections. Start with the most obvious group that has the least ambiguity, and then your problem space is smaller on category 2, then the same for category 3 and 4.

So really you could fine tune 3 models - one for 16 words, one for 12, and one for 8. Then use them in succession.

Also, if you come across a mistake at the end (have some negative examples in the training sets), tell it to start over and add to the prompt what you think is NOT a group.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#30

Post by s1mon »

I spent a bunch of time manually just using GPT4 with fairly simple prompts and giving it the same feedback that the game gives. There's an archive of puzzles which I used to try to train it with, and sometimes it would be very successful, and sometimes it was frustrating how bad it was at doing basic things like keeping track of what words it had used so far. Each day I would also have it play the new puzzle from the NYTimes which it couldn't have trained on. Some days it did perfectly some it made really stupid mistakes. It seems like a more concerted effort could achieve better results.
Post reply on HN