61–70 of 361 posts
Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.
"Penalty for Illegal Moves The penalty for the first illegal move is to add two minutes to the opponent's time. A second illegal move results in loss of the game.".
https://new.uschess.org/sites/default/files/media/documents/...
Earlier quoted context omitted.
Again, not enough for 1400. 800 maybe.
We have evidence right here that patterns of moves leads to 1400. To me this is evidence that 1400 level play is very rigid and mostly relies on defined openings and strategies, making it easy for an LLM to find similar games and win using standard continuations. Just like grammar the patterns are too hard for humans to see and encode, but LLMs can encode pretty complex patterns. Domains that are easy to encode as gr…
Earlier quoted context omitted.
Nobody who is 1400 plays outright illegal moves.
Yet it achieves 1400. Add hard rules to stop it spewing out said moves and you have a 1400 ELO Player (most UIs won't even let you make illegal moves). It is difficult to say that is not impressive due to it being an emergent ability.
I don't know why you think it's an emergent ability.
It's seeing a sequence of moves, and playing the most likely next move (i.e. the most likely next token) given the previous complete move sequences it was trained on. That's the baseline of what an LLM does—not something emergent. Games in online chess databases tend to be of relatively good players. Nobody wants to look up games played by two 800 ELO players.
As an aside, there have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here.
Earlier quoted context omitted.
> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?
If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.
I tried it with this prompt and it works a little bit but produces repeated illegal moves as early as move 4.
Bummer. Perhaps GPT-4 will be more capable.
Earlier quoted context omitted.
It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.
Perhaps I don’t know what “easy” means in your comment, but the OP article says that they “almost always” played full, legal games with their prompt.
I don't know why it worked in this specific case, but based on earlier examples it is more likely that these kind of games were more prevalent in its dataset it was trained on than it being able to play chess in general. It still wasn't perfect, so even these games weren't rigid enough for it to reliably perform valid moves.
Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.
And a good hacker can add these features to Alpaca and run it on his desktop...
Import all his WhatsApp chats and private emails into the corpus.
Add speech recognition and TTS.
Better than me then. But does it give credit to who taught it. These models are basically a scrape of the best of humankind and a claim that it's their own.