Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

91–100 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#91

Post by dudeinjapan »

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

Obviously the article should be taken with a giant grain of salt. That being said, not many things what aren't designed to play chess can play chess, with or without coaxing. My dog cannot, for instance, nor can my coffee table.

Re: ChatGPT's Chess Elo is 1400

#92

Post by qwytw »

johnfn wrote: post #50
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

Is it though? I mean if you had data on millions of games what is the chance that you'd find one which has identical position that the one you're in (it's not like most moves are random..)

I wonder how well it could perform in Go, there are way more permutations there so finding an identical state should be more difficult.

Re: ChatGPT's Chess Elo is 1400

#93

Post by ballenf »

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

The illegal moves were counted as losses/resignations, not ignored.

Re: ChatGPT's Chess Elo is 1400

#94

Post by __s »

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

> So whenever it wanted to make an illegal move, it resigned.

Doesn't sound like ignoring the cases where it failed

Re: ChatGPT's Chess Elo is 1400

#95

Post by vidarh »

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

Fuller context from the article:

> Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned.

(my emphasis)

So the illegal moves are at least part of the reasons for the 6 losses, and factored into the rating. Quickly scanning the game, it seems 3 of the losses ended in checkmate, so that leaves 3 illegal moves in 19 games.

Could be better, but for a system not intentionally built to play chess, it's pretty decent.

Re: ChatGPT's Chess Elo is 1400

#97

Post by henryfjordan »

chpatrick wrote: post #75
henryfjordan wrote: post #54

Earlier quoted context omitted.

It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.

And the 99% probability isn't an evidence of understanding chess?

I don't know.

Part of me wants to say no, that the model "thinks" in terms of text it has seen and so knows from chess forums it has seen that certain text representing moves come naturally after previous moves' text. It doesn't understand anything other than certain text comes after other text.

But yeah at the same time I can see how it is thinking inside the world we built for it. We have senses like touch, smell, sight. The only "sense" these models have are an input text box. Would we even necessarily recognize intelligence when it is so different from our own?

So does it understand chess like I do? No, it cannot. Does it understand chess at all? I'm not sure. I'm not sure I'd understand chess in it's world either though.

Re: ChatGPT's Chess Elo is 1400

#98

Post by qwytw »

psychphysic wrote: post #85
latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

That's how one uses any tool.

The behavior of pretty much every other tool is much easier to interpret though.

Re: ChatGPT's Chess Elo is 1400

#99

Post by Andrew_nenakhov »

I tried playing with it and it barely makes any correct move: forgets whos turn to move, doesn't observe the rules, incorrectly displays move results.

So I guess that if we ask it to make a move repeatedly and accept the move only when it occasionally makes a legal move, it'll show some results, but I don't think it is what can be interpreted as a chess engine with some meaningful rating.

Re: ChatGPT's Chess Elo is 1400

#100

Post by Out_of_Characte »

psychphysic wrote: post #85
latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

That's how one uses any tool.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.
Post reply on HN