I own't speak to weather this shows ChatGPT can play chess, but did you notify your opponents beforehand they are playing chatgpt? Otherwise (maybe even if) you're cheating and your account should be removed. If anyone else here is thinking of doing this - don't.
ChatGPT's Chess Elo is 1400
131–140 of 361 posts
Re: ChatGPT's Chess Elo is 1400
#132I just opened a random recent chess game on lichess ( https://lichess.org/YpxTUUbO/white#88 ) . I'm pretty sure ChatGPT can't be trained on games that were just played, so this ensures the game is not in its training data. I gave the position before checkmate to ChatGPT to see if it would produce the checkmating move. It played an illegal move, replying with "Be5#" even there's no bishop of either color in the positi…
I gave it this prompt, and pasted the game. Its reply:
"45. Qxg6# This is a checkmate!"
Re: ChatGPT's Chess Elo is 1400
#133Elo 1400 seems to equate to a top 30% player. Source: https://chessgrandmonkey.com/chess-rating-percentile-calcula...
slight nit: this is true amongst all chess players. amongst professionals and amateurs, 1400 is technically Class C - which is on the lower end of amateurs. nonetheless, it's cool to see a model 'generally intelligent' enough to beat most players. what's more interesting to me is that according to OP's analysis, GPT4 is worse at Chess than GPT3.5... source: https://en.wikipedia.org/wiki/Chess_rating_system
It would depend how often did this happen:
> Occasionally it does make an illegal move
Since it would lose every game in which it made more than a single illegal move.
Re: ChatGPT's Chess Elo is 1400
#134> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…
Is this the top comment (and not even grey) because more people failed to read the article than read it?
Re: ChatGPT's Chess Elo is 1400
#135Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
> If it had any understanding of chess, it would never play an illegal move.
Plenty of people who have a basic understanding of chess would make an illegal move if they had no board to see and had to play only with notation.
Re: ChatGPT's Chess Elo is 1400
#136Earlier quoted context omitted.
raw statistical power.
a game of chess becomes « new » after a few moves. starting middlegame, you’re in unknown territories and have no statistics to refer to..
I'm referring to the statistical power of the model. For example, if you replace GPT4 with GPT2 it will lose every game, because the statistical power is lower. Increasing the statistical power doesn't make the model understand any better, it just makes it more likely to generate a response that aligns with human expectations.
Re: ChatGPT's Chess Elo is 1400
#137Earlier quoted context omitted.
At least it should make valid moves, that is the minimum level required. It didn't reach that level here. If it never made illegal moves we could talk and see what it does, but until then we can be sure it didn't understand the rules.
I don’t understand why the threshold is “never”. Isn’t it entirely possible that the AI is learning a model of chess but this model is imperfect? What if AIs don’t fail the same way as humans?
> What if AIs don’t fail the same way as humans?
But it is failing the same way as a human. Humans who remembers patterns and don't learn the logic makes these kind of errors in math or logic all the time.
ChatGPT is much better than humans at pattern matching, you see it right here it can pattern match chess moves and win games! But its inability to apply logic to its output instead of just pattern matching is holding it back, as long as that isn't solved it wont be able to perform on the level of humans in many tasks. Chess might be easy enough to solve using just pattern matching and no logic that scaling it up will make it pretty good at chess, but many other topics wont be.
Re: ChatGPT's Chess Elo is 1400
#138Calling it: chess games in the future will be a battle between two prompts.
Well we already had much better chess bots years ago to be fair.. People still play chess for some reason (not sure why)
Re: ChatGPT's Chess Elo is 1400
#139This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level. It may then reach superhuman level in any ta…
According to https://chess.stackexchange.com/questions/2550/what-are-the-... median rating is 1148 (252,989 Players). So it's beating half of humanity at a mind sport and it wasn't even specifically trained for it.
There's a difference between chess.com and USCF ratings.
Re: ChatGPT's Chess Elo is 1400
#140Earlier quoted context omitted.
> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.
Is it though? I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..) I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.
You can click through the Lichess opening database (click the book icon, and then the Lichess tab) to get an idea: https://lichess.org/analysis
But the answer is insanely unlikely, past a certain number of moves. The combinatorial explosion is inescapable. Even grandmaster games are often novelties in So, it has a to have some kind of internal representation of board state and what makes a reasonable move and such that enables it to generalize (choosing random legal moves is almost unbelievably bad, so it’s not doing that).
I also doubt that it has been trained on the full (massive) database of Lichess games, but that would be an interesting experiment: https://database.lichess.org/