Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

181–190 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#181

Post by imtringued »

circuit10 wrote: post #14
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

I wouldn’t say it has no understanding whatsoever, it probably just doesn’t have the capability to keep track of everything that happened all the time

When I ask it to display moves in an ASCII board it messes up. I doubt that having a representation of the board is the key to improving its chess skills.

Re: ChatGPT's Chess Elo is 1400

#182

Post by Pxtl »

swatcoder wrote: post #144
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

No ELO 1400 player will have that rate of illegal moves, so saying it that it plays with an ELO 1400 rating is disingenuous. Reinterpreting illegal moves as resignation is absurd when an LLM is formally capable of expressing statements "I resign" or "I cannot conceive of a winning move from here" just as well as any human player. It just doesn't do so because it's not actually playing chess the way we think of an ELO…

I'd be interested if it could be coaxed into legal moves after making an illegal one. "That is an illegal move. Can you do something legal with this board?"

Re: ChatGPT's Chess Elo is 1400

#184

Post by CSMastermind »

mtlmtlmtlmtl wrote: post #175

There's a huge difference between 1400 elo in FIDE games versus 1400 on chess.com, which is not even using elo. For instance the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating. Chess.com and lichess have a ton of rating inflation.

> the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating

Online rating inflation is real but I'm not sure blitz is the best example of it because in that case there is a notable difference between online and otb (having to take time to physically move the pieces).

Re: ChatGPT's Chess Elo is 1400

#185

Post by LeanderK »

sebzim4500 wrote: post #155
LeanderK wrote: post #128

Earlier quoted context omitted.

I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.

I'm sure the MuZero chess policy network would reach much higher than 1400, and that has no notion of recursion either. (And also wasn't taught the rules explicitly)

Yes, I wouldn't say it's impossible, but that's just a hard limit from the architecture. MCTS would have to terminate after a few iterations. And the number of steps is not that big, just massively parallel.

Re: ChatGPT's Chess Elo is 1400

#186

Post by johnfn »

mtlmtlmtlmtl wrote: post #110
johnfn wrote: post #50

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not. These series of moves are referred to as opening theory and are described at copious length in tons of books. This is because while the raw number of possible p…

I'm very familiar with opening theory. Some of the games are 40 or 60 movies. This is not a regurgitation of book moves.

Re: ChatGPT's Chess Elo is 1400

#187

Post by Jensson »

oezi wrote: post #130
WoodenChair wrote: post #63

Earlier quoted context omitted.

> It is difficult to say that is not impressive due to it being an emergent ability. I don't know why you think it's an emergent ability. It's seeing a sequence of moves, and playing the most likely next move (i.e. the most likely next token) given the previous complete move sequences it was trained on. That's the baseline of what an LLM does—not something emergent. Games in online chess databases tend to be of relat…

> here have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here. It could be, but would you think that of the 100-300 bn parameters in the model a lot are dedicated to chess move sequences? It seems likely that it has seen such data, but I would be surprised if it is using…

The web has millions of grandmaster chess games and probably billions of chess games overall. So I wouldn't be surprised if it has like 0.01% to chess games since there are so many, if so that would mean it has 10-30 million parameters to play chess with, for comparison stockfish has 10 million parameters to its chess engine.

Re: ChatGPT's Chess Elo is 1400

#188

Post by kmeisthax »

saghm wrote: post #157
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

So it sounds like it can play _some_ legal chess games, but not all; it's unable to consistently complete a game where it loses. Maybe the remaining work shouldn't be focused on trying to teach it chess rules better, but to teach it sportsmanship better. People were so excited about teaching it high-school level academics that we forgot to teach it the basic lessons we learn in kindergarten.

Or append "If you wish to resign or you cannot think of a legal move, type 'resign'" to the end of the prompt.

Re: ChatGPT's Chess Elo is 1400

#189

Post by saghm »

vidarh wrote: post #171
saghm wrote: post #157

Earlier quoted context omitted.

So it sounds like it can play _some_ legal chess games, but not all; it's unable to consistently complete a game where it loses. Maybe the remaining work shouldn't be focused on trying to teach it chess rules better, but to teach it sportsmanship better. People were so excited about teaching it high-school level academics that we forgot to teach it the basic lessons we learn in kindergarten.

It seems like it plays mostly legal chess games, when not explicitly reminded of the rules. There's no problem of sportsmanship when it makes mistakes in a game it has not been verified to understand the rules of.

I was responding to the conclusion from TFA quoted by the parent comment, that playing an illegal move was it saying "this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” If you reject that premise, then yes, my response to it will not be particularly relevant to your worldview.

Re: ChatGPT's Chess Elo is 1400

#190

Post by andrepd »

> Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned.

Lmao. ChatGPT can play chess, as long as you ignore the times when it can't x) Brilliant.

Post reply on HN