Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

121–130 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#121

Post by sebzim4500 »

mtlmtlmtlmtl wrote: post #110
johnfn wrote: post #50

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not. These series of moves are referred to as opening theory and are described at copious length in tons of books. This is because while the raw number of possible p…

Yeah but now explain how it played a 61 move game.

EDIT: I checked and it left the lichess database after 9 moves. The lichess db has probably 5 orders of magnitude more chess games in it than chatGPT has in its training data.

Re: ChatGPT's Chess Elo is 1400

#122

Post by LudwigNagasena »

xapata wrote: post #59
LudwigNagasena wrote: post #41

Earlier quoted context omitted.

That’s a bit meaningless without implied population. And it doesn’t tell you much because it depends so much on the particulars (how many years people in that group spent on chess, how serious they are). I would guess that’s quite different for FIDE and online chess. Intuitively, I would guess 1400 is the rating of an average kid who spent a year studying chess seriously.

Not the ELO of the average kid who spends a year studying chess seriously? It's very difficult to define the average kid without context. It's been a while since I read a global demographics update, but not all that long ago the mode child would have been living on $2 a day.

Sure, it’s tricky, but I feel it’s more coherent than “all chess players”.

Re: ChatGPT's Chess Elo is 1400

#124

Post by qwytw »

wahnfrieden wrote: post #45
jonathanstrange wrote: post #7

Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.

ai isn't to be used completely raw as text. you can ask gpt to write a python program that filters its responses for legal moves. then it's a matter of rendering the output (immediate execution) in a richer format than raw text.

I did. It write 15-20 lines and then finished the functions with this:

" # Add more cases for other piece types such as rooks, knights, bishops, queens, and kings. # ... "

So not great..

But even if it wasn't so lazy it would just find a script (or multiple scripts) which say that they do that. Probably combine them or add some minor stuff and output the result.

It can't really verify that data it used to come up with this result is accurate or reason about it in any way? Can it?

So if you're incapable of verifying that yourself it would be pretty useless?

Re: ChatGPT's Chess Elo is 1400

#125

Post by wongarsu »

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

Pattern matching on databases of previous moves is a legitimate technique that was poplar in chess AI back when computing power was more limited. Even if that was all that GPT did here it would still be interesting.

But the presence of illegal moves doesn't really show that in my eyes. I fully understand the rules of chess, but I still occasionally make illegal moves. In 2017 Magnus Carlsen made one in a tournament [1]. The number of illegal moves suggests that either GPT is pretty new to chess, has low intelligence, or is playing under difficult circumstances (like not having a chess board at hand to keep track of the current state). I'm not sure we can deduce more than that

1: https://www.chessbase.in/news/Carlsen_Inarkiev_controversy

Re: ChatGPT's Chess Elo is 1400

#126

Post by medvezhenok »

dragontamer wrote: post #104
oezi wrote: post #86

Earlier quoted context omitted.

I think there is very low percentage of players at elo 1400 who can provide a valid next move after seeing just the list of moves and not the current board state.

I'm Elo 1400 and can beat literally everyone I know in the real world. I need to go online to find players at my skill level, or find tournament/competitive settings for a challenge. Yeah, I'm "class C", weak amateur chess player, but I think you're grossly underestimating the amount of study I put into this game. I'm not going to make an illegal move

You will under time pressure :) even Grandmasters have done that (I'm around 1900 elo for context)

Also people forgetting they moved the king/rook and trying to castle.

Re: ChatGPT's Chess Elo is 1400

#127

Post by lb4r »

xapata wrote: post #70
lb4r wrote: post #65

Earlier quoted context omitted.

You can still have syntactically correct moves that are illegal.

Sorry, I meant that learning how to speak with syntax and grammar is comparable to a string of mostly-correct chess moves. Especially in early game.

Ah, yes that is more or less my understanding of it as well. Though I would like to see how it would perform if given the state of the board as input to predict the next move, rather than a sequence of moves, since that is how we humans normally determine the next move. I believe the move history is only relevant when it comes to en passant and certain draw scenarios (like repetition and that 50 move rule). Needless to say, it would first have to be trained on those types of inputs, which it probably is not.

Edit: move history can also be relevant when it comes to castling.

Re: ChatGPT's Chess Elo is 1400

#128

Post by LeanderK »

marcodiego wrote: post #79

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level. It may then reach superhuman level in any ta…

I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.

Re: ChatGPT's Chess Elo is 1400

#129

Post by hgsgm »

Out_of_Characte wrote: post #100
psychphysic wrote: post #85

Earlier quoted context omitted.

That's how one uses any tool.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.

All this above, and people are claiming that ChatGPT lacks human level comprehension of the text it consumes.

In Chess.com, you absolutely can attempt an illegal move, and many players do, and you will not get punished for it, so chatgpt is better then a 1400 human player.

Re: ChatGPT's Chess Elo is 1400

#130

Post by oezi »

WoodenChair wrote: post #63
mrbungie wrote: post #47

Earlier quoted context omitted.

Yet it achieves 1400. Add hard rules to stop it spewing out said moves and you have a 1400 ELO Player (most UIs won't even let you make illegal moves). It is difficult to say that is not impressive due to it being an emergent ability.

> It is difficult to say that is not impressive due to it being an emergent ability. I don't know why you think it's an emergent ability. It's seeing a sequence of moves, and playing the most likely next move (i.e. the most likely next token) given the previous complete move sequences it was trained on. That's the baseline of what an LLM does—not something emergent. Games in online chess databases tend to be of relat…

> here have been chess programs for years that show you for a given position all of the previous games in its database with the same position and the win outcome % of each move. That's all that's going on here.

It could be, but would you think that of the 100-300 bn parameters in the model a lot are dedicated to chess move sequences? It seems likely that it has seen such data, but I would be surprised if it is using a considerable chunk to store chess database information.

Post reply on HN