Good thing it's "incapable of reasoning"!
It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.
How did it win 11 out of 19 games then, blind luck?
ChatGPT's Chess Elo is 1400
21–30 of 361 posts
Re: ChatGPT's Chess Elo is 1400
#22Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
You don’t get to 1400 like that. The amount of moves it has to literally remember is stupendous.
Nobody who is 1400 plays outright illegal moves.
Re: ChatGPT's Chess Elo is 1400
#23Re: ChatGPT's Chess Elo is 1400
#24Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.
That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.
Re: ChatGPT's Chess Elo is 1400
#25Good thing it's "incapable of reasoning"!
Is a normal chess program capable of reasoning?
I would say so. Logical reasoning is not a hard thing to program a computer to do: the really hard part is to reason about the real world, rather than just a manually-defined world model.
Traditional algorithms are vanguards of pure reason. Neural networks are super-intuition. Stockfish unites the two, executing an efficient intuitive heuristic search over the solution space of “moves in a chess game” – but no matter how cool the robot arm you build for it, Stockfish could never learn to assemble furniture.
Re: ChatGPT's Chess Elo is 1400
#26Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
I wouldn’t say it has no understanding whatsoever, it probably just doesn’t have the capability to keep track of everything that happened all the time
that's also not entirely true either. it's prone to make illegal moves. one could argue abiding by the rules of the game is a requisite for understanding!
Re: ChatGPT's Chess Elo is 1400
#27Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
I would argue that a decent human player also has "similar sequences of moves in its training set", due to reading plus playing or watching games.
The illegal move argument is good though, and indicates no direct understanding of what it is spewing out.
Re: ChatGPT's Chess Elo is 1400
#28When I tried it at v3.0 i found after 5-10 moves it started moving illegally.
The AI has simply, and correctly, identified that cheating is the best way to win at something.
It doesn't even know the rules, let alone cheat. It predicts the notation from the massive amount of games seen during training.
Edit: although thinking of it, it probably anazyled a shitload of chess books too. It might have a lot of knowledge compressed into the internal representation. So yeah, maybe it knows rules in some form and even some heuristics, after all. It just doesn't understand the importance of making legal moves, and can't apply complex theory (requiring it to run stateful algorithms).
Re: ChatGPT's Chess Elo is 1400
#29Re: ChatGPT's Chess Elo is 1400
#30Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
You don’t get to 1400 like that. The amount of moves it has to literally remember is stupendous.
It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.