ChatGPT's Chess Elo is 1400 • Hacker News Board

Re: ChatGPT's Chess Elo is 1400

#81

Post by fxtentacle » Fri, Mar 17, 2023, 6:14 PM UTC

johnfn wrote: post #50

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

For Bomberland, we were quite surprised how strongly we could compress and quantize the current game state and still get useful movement predictions.

I wouldn't be surprised if the relevant state in a typical beginner's chess game also excluded many units in the sense that yes, you could move them, but a beginner is going to just ignore them in any case.

Re: ChatGPT's Chess Elo is 1400

#82

Post by fsckboy » Fri, Mar 17, 2023, 6:14 PM UTC

johnfn wrote: post #50

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

>playing a move from a different game in a new game is nonsensical

GP did say "sequence of moves", and if it matches what it has seen from the first move on, including the opponent, it will be in a valid "sequence of moves".

then, even midgame or endgame, if a sequence is played on one side of the board, even though the other side of the board may be different, the sequence has a great chance of being good (not always of course, but a 1400 rating is solid (you know the rules and some moves) but not amazing

Re: ChatGPT's Chess Elo is 1400

#83

Post by rzzzt » Fri, Mar 17, 2023, 6:14 PM UTC

mjr00 wrote: post #9

juice_bus wrote: post #5

When I tried it at v3.0 i found after 5-10 moves it started moving illegally.

The AI has simply, and correctly, identified that cheating is the best way to win at something.

Tom 7's NES play function paused the game when it encountered an insurmountable problem: https://youtu.be/xOCurBYI_gY?t=950

Re: ChatGPT's Chess Elo is 1400

#84

Post by Jensson » Fri, Mar 17, 2023, 6:15 PM UTC

famouswaffles wrote: post #64

WoodenChair wrote: post #40

Earlier quoted context omitted.

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

Not if training is unsupervised. If you've never been explicitly told the rules of game, you can never be 100% sure of all possible illegal moves. anyway the 3.5 series can't ply chess but gpt-4 certainly can.

> anyway the 3.5 series can't ply chess but gpt-4 certainly can.

This article stated the opposite, gpt-4 couldn't play chess while gpt-3.5 could. So this is a case where the model got dumber.

Re: ChatGPT's Chess Elo is 1400

#85

Post by psychphysic » Fri, Mar 17, 2023, 6:16 PM UTC

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

That's how one uses any tool.

Re: ChatGPT's Chess Elo is 1400

#86

Post by oezi » Fri, Mar 17, 2023, 6:16 PM UTC

WoodenChair wrote: post #40

PoignardAzur wrote: post #35

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

I think there is very low percentage of players at elo 1400 who can provide a valid next move after seeing just the list of moves and not the current board state.

Re: ChatGPT's Chess Elo is 1400

#87

Post by melenaboija » Fri, Mar 17, 2023, 6:16 PM UTC

johnfn wrote: post #50

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

If it isn not memorizing, how do you think is doing it?

Re: ChatGPT's Chess Elo is 1400

#88

Post by baq » Fri, Mar 17, 2023, 6:16 PM UTC

Longlius wrote: post #15

chpatrick wrote: post #6

Good thing it's "incapable of reasoning"!

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

How do you know that? It has billions of parameters, some of them may well be for internal understanding of chess?

Re: ChatGPT's Chess Elo is 1400

#89

Post by serverholic » Fri, Mar 17, 2023, 6:16 PM UTC

Jensson wrote: post #74

SanderNL wrote: post #62

Earlier quoted context omitted.

At what point can we just say that understanding “patterns of moves” is understanding chess? It seems you suggest there is more to it, but maybe I am mistaken.

At least it should make valid moves, that is the minimum level required. It didn't reach that level here. If it never made illegal moves we could talk and see what it does, but until then we can be sure it didn't understand the rules.

I don’t understand why the threshold is “never”. Isn’t it entirely possible that the AI is learning a model of chess but this model is imperfect? What if AIs don’t fail the same way as humans?

Re: ChatGPT's Chess Elo is 1400

#90

Post by baq » Fri, Mar 17, 2023, 6:17 PM UTC

henryfjordan wrote: post #54

sebzim4500 wrote: post #20

Earlier quoted context omitted.

Mostly it didn't make illegal moves though, since illegal moves mean resignation and it won more than it lost. Making 60 legal moves in a row in one game would be the coincidence of the century unless it had some knowledge of the rules of chess.

It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.

Markov chains are probabilistic text models and rather far from 1400 elo