Live data from Hacker News

(Unsuccessfully) Fine-tuning GPT to play "Connections"

danielcorin.com

11–20 of 54 posts

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#11
post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

Néophyte question:

Can we infer anything about what llm's can achieve from what we can achieve with AIs like AlphaGo? I thought their approaches were completely separated

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#12

I unfortunately can’t imagine having time to test this, but I imagine there may be a way to accomplish this with embeddings. The game itself is sort of an embeddings clustering problem, with the added difficulty that each group needs to only be alike in 1 way (versus a full vector distance which measures how alike they are in every way). Maybe there is some way to search for a vector of weights, which, when multiplie…

[deleted]

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#13

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.

>Why can't you click & drag the words to reorder them? That level of difficulty is part of the game. There is a shuffle button to ease the ideas generation but most likely is was done like that by design.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#14
You’ll probably get better results by putting the examples only in the completion part of the training examples.

GPT-3.5 learns how to generalize better when it’s just in the completion.

This is the same problem that vexed the researchers who did the paper on the alleged reversal curse.

(https://andrewmayne.com/2023/11/14/is-the-reversal-curse-rea...)

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#15
I suspect the inability of the model to "plan ahead" is a significant contributor to its poor performance relative to a human. Being able to check a grouping to be sure it includes at least four words _and_ to check that it doesn't conflict with the other three groupings is a major advantage - it's pretty common that these puzzles include partial or incompatible red herring groups.

If this is the case, performance might be improved by taking the final solving responsibility away from the model and giving it to the script. You could ask GPT for categories, ask whether each word fits each category (discarding categories with fewer than 4 words), and then search for 4 non-overlapping categories.

(This might be missing the point of the exercise though.)

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#17
post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

> This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well

I would speculate it’s struggling because of the linear nature of its output, and the red-herring words which crossover between categories.

Because the model can’t “look ahead”, it starts spitting out valid combinations, but without being able to anticipate that committing to a certain combination early on will lead to a mistake later.

I expect if you asked it to correct its output in a followup message, it could do so without much difficulty.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#18

This game looks cool but wow the UX is terrible. Why can't you click & drag the words to reorder them? Seems half the difficulty is keeping track of your thought process with the inability to make a draft state.

Also, there is no reason to limit the amount of guesses. Just let me try until I figure it out. But no, they've put a limit so that it can be "sharable" in a tweet-sized text, to try to copy the viralness of Wordle. But they do it to the detriment of the gameplay, in such a way that I don't even bother playing.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#19

This is pretty interesting. Intuitively, Connections is the kind of thing I would expect GPT to not be good at, because almost every day there's something that feels kind of "out of left field" in the categories. In my experience LLMs are good at regurgitating the "standard" take on a topic, or "best practices", but lack the creativity and out-of-the-box thinking that makes Connections fun. On the other hand, it feel…

The "whole point" of embeddings is that words have a vector that represents how well that word fits into a certain categories, so words belonging together is close in that vector space. So in that sense it almost feels like this should be solvable using something simpler than a full LLM. To "just" get the embeddings of the words, and then find the groups of 4 that minimizes the total distances within the groups.

Re: (Unsuccessfully) Fine-tuning GPT to play "Connections"

#20
post #6

This game is well known in the UK as the "Connecting Wall" from Only Connect. This result - poor Chat GPT performance - surprises me. I thought pattern detection and set forming was something that Chat GPT could do well. Perhaps it would need a model to be specifically trained for this task. If alpha-zero can master chess, then surely this game isn't beyond what is trainable. You can prompt Chat GPT that it'll be pla…

Néophyte question: Can we infer anything about what llm's can achieve from what we can achieve with AIs like AlphaGo? I thought their approaches were completely separated

Not really;

Gpts are a class of text predictors. Ultimately they are ranked on whether or not the output is similar to the training data, text-wise. If the training data included a game then it may be able to play that game, but only if that game requires reasoning about entire words (because of tokenization, gpts can't reason in terms of letters, that's why they do poorly at crosswords for example)

On the flip side, alphazero is a class of networks that have a list of actions they can take, and a list of parameters they observe about the game (in chess: the board position, in other games: their position on screen, score, speed, etc). The model is then trained to take actions that maximize an actual hard value from the game, like winning a game of chess, capturing a piece, increasing a score, driving the furthest.

In theory you could train a model with the alphago method to do text prediction, but LLMs are called "large" for a reason, the input and output spaces would have to be the number of possible tokens (and at that point just train a normal gpt, it's much more efficient). Also in theory you could train a gpt to play games, but you're spending huge amounts of compute evaluating extraneous words in the input (the prompt) and the output (most words do not have anything to do with your game). on top of that, you're iterating over every word you generate to generate the next one, so you're doing multiple passes of this largely infficient computing, which means you're slower compared to a tailor-made model that can evaluate one situation once and give you a list of outputs to perform.

in this specific case it's a bit wierd because the input space for the alphazero model would have to be every word that can appear on the board, but the reasoning part is most likely not a problem given enough model size. since it's competing with a multi-gigabyte llm though, there is space to spare.

Post reply on HN