Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

161–170 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#161

Post by sumtechguy »

the_af wrote: post #148

This is GPT4, right? Because ChatGPT (GPT-3) still fails to provide a legal game of Tic Tac Toe with this prompt: > "Let's play Tic Tac Toe. You are O, I'm X. Display the board in a frame, with references for the axes" It failed to recognize that I won. Then continued playing (past the end), played illegally over a move I had already done, obtained a line of 3 for itself, and still doesn't acknowledge the game has en…

For me when I tried it, it forgot the board half way thru and decided there were no more moves possible (4 left). Then forgot one of my moves.

It is an NLM it does not have the ability to reason. In my case the NLM 'next iteration' was a half a game with a missing move and 'nothing left to do'.

Re: ChatGPT's Chess Elo is 1400

#163

Post by RyanCavanaugh »

Longlius wrote: post #15
chpatrick wrote: post #6

Good thing it's "incapable of reasoning"!

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?

Re: ChatGPT's Chess Elo is 1400

#164

Post by sapiogram »

gfd wrote: post #115
marcodiego wrote: post #79

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level. It may then reach superhuman level in any ta…

According to https://chess.stackexchange.com/questions/2550/what-are-the-... median rating is 1148 (252,989 Players). So it's beating half of humanity at a mind sport and it wasn't even specifically trained for it.

That's USCF ratings, chess.com ratings are massively inflated in comparison.

Re: ChatGPT's Chess Elo is 1400

#165

Post by medvezhenok »

dragontamer wrote: post #158
medvezhenok wrote: post #156

Earlier quoted context omitted.

The article points out that the way that game was conducted was bad. (Here's the original transcript: https://pastebin.com/X6kBRTa9 ) You need to give ChatGPT the full state (every move) on every prompt to make it play closer to 1400. The game you linked the user was giving one move at a time.

> You need to give ChatGPT the full state If I've been given the full state every move, I will _never_ make an illegal move as a 1400 chess player. ----------- > O-O > I'll play O-O as well. Your move. Do you really think that this error would have been made at 1400 Elo? Even in blind chess? This is the 5th move of the game. I can still track the game at this point mentally. I recognize that you're 1900 and think tha…

And yet kids who are gaining rating quickly can and do still occasionally (albeit rarely) make illegal moves at 1400. I know because I've played them (and was one, many years ago).

Re: ChatGPT's Chess Elo is 1400

#166

Post by sebzim4500 »

qwytw wrote: post #138
jackphilson wrote: post #3

Calling it: chess games in the future will be a battle between two prompts.

Well we already had much better chess bots years ago to be fair.. People still play chess for some reason (not sure why)

Are you confused at the idea that someone might do something for fun?

Re: ChatGPT's Chess Elo is 1400

#167

Post by vidarh »

swatcoder wrote: post #144
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

No ELO 1400 player will have that rate of illegal moves, so saying it that it plays with an ELO 1400 rating is disingenuous. Reinterpreting illegal moves as resignation is absurd when an LLM is formally capable of expressing statements "I resign" or "I cannot conceive of a winning move from here" just as well as any human player. It just doesn't do so because it's not actually playing chess the way we think of an ELO…

That no human ELO 1400 player will have that rate of illegal moves may be true, but if anything treating the very first illegal move as forfeit appears to be stricter than most rules

Re: ChatGPT's Chess Elo is 1400

#169

Post by ncallaway »

Out_of_Characte wrote: post #100
psychphysic wrote: post #85

Earlier quoted context omitted.

That's how one uses any tool.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.

> almost all people from ~300 elo can play without making illegal moves

I don't believe you. Are you giving those people a restricted move set (i.e. computer chess, where it will _only_ allow legal moves)? Because if you give people an unrestricted board, I _guarantee_ you people will make lots of illegal moves.

Me: Moves pawn

Opponent: You can't do that, you exposed your king to check.

Me: Oops, sorry, you're right.

Re: ChatGPT's Chess Elo is 1400

#170

Post by chpatrick »

root_axis wrote: post #136
bsaul wrote: post #80

Earlier quoted context omitted.

a game of chess becomes « new » after a few moves. starting middlegame, you’re in unknown territories and have no statistics to refer to..

I'm referring to the statistical power of the model. For example, if you replace GPT4 with GPT2 it will lose every game, because the statistical power is lower. Increasing the statistical power doesn't make the model understand any better, it just makes it more likely to generate a response that aligns with human expectations.

"Statistical power" isn't some magic property of GPT4. It can produce statistically more likely moves because somewhere deep down it can model chess.
Post reply on HN