Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

171–180 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#171

Post by vidarh »

saghm wrote: post #157
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

So it sounds like it can play _some_ legal chess games, but not all; it's unable to consistently complete a game where it loses. Maybe the remaining work shouldn't be focused on trying to teach it chess rules better, but to teach it sportsmanship better. People were so excited about teaching it high-school level academics that we forgot to teach it the basic lessons we learn in kindergarten.

It seems like it plays mostly legal chess games, when not explicitly reminded of the rules. There's no problem of sportsmanship when it makes mistakes in a game it has not been verified to understand the rules of.

Re: ChatGPT's Chess Elo is 1400

#172

Post by yieldcrv »

the_af wrote: post #148

This is GPT4, right? Because ChatGPT (GPT-3) still fails to provide a legal game of Tic Tac Toe with this prompt: > "Let's play Tic Tac Toe. You are O, I'm X. Display the board in a frame, with references for the axes" It failed to recognize that I won. Then continued playing (past the end), played illegally over a move I had already done, obtained a line of 3 for itself, and still doesn't acknowledge the game has en…

No, it is GPT 3.5

and the real answer to every debate about capabilities comes down to "you're prompting it wrong".

Re: ChatGPT's Chess Elo is 1400

#173

Post by div72 »

wongarsu wrote: post #125
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

Pattern matching on databases of previous moves is a legitimate technique that was poplar in chess AI back when computing power was more limited. Even if that was all that GPT did here it would still be interesting. But the presence of illegal moves doesn't really show that in my eyes. I fully understand the rules of chess, but I still occasionally make illegal moves. In 2017 Magnus Carlsen made one in a tournament […

I agree with your conclusion but the example you gave of Magnus Carlsen doing an illegal move is a pretty bad one imo. That was clearly a trap from his opponent as he did an illegal move first by checking Magnus while he was under check and then immediately calling the arbiter when Magnus moved his king in response. Magnus also probably did not have had much time left as it was a Blitz game.

Re: ChatGPT's Chess Elo is 1400

#174

Post by JellyBeanThief »

swatcoder wrote: post #144
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

No ELO 1400 player will have that rate of illegal moves, so saying it that it plays with an ELO 1400 rating is disingenuous. Reinterpreting illegal moves as resignation is absurd when an LLM is formally capable of expressing statements "I resign" or "I cannot conceive of a winning move from here" just as well as any human player. It just doesn't do so because it's not actually playing chess the way we think of an ELO…

Fine, just wrap the LLM in a simple function that detects illegal moves and replaces them with "I resign" or "I cannot conceive of a winning move from here". Then you aren't "reinterpreting" anymore.

My point is, it sounds like Elo doesn't measure what we want it to measure. If we care about the way an agent wins a game and not just whether it wins a game, then we need an instrument that measures strategy, not outcome.

Re: ChatGPT's Chess Elo is 1400

#175

Post by mtlmtlmtlmtl »

There's a huge difference between 1400 elo in FIDE games versus 1400 on chess.com, which is not even using elo.

For instance the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating. Chess.com and lichess have a ton of rating inflation.

Re: ChatGPT's Chess Elo is 1400

#176

Post by LeanderK »

mrguyorama wrote: post #146
LeanderK wrote: post #109

Earlier quoted context omitted.

by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.

It cannot be learning the general rules because it occasionally tries to invent pieces out of whole cloth.

that would be the optimal rule :) We usually optimise to improve and learn better and better rules, hopefully approximating the optimal rule after some iterations. There's still a gap here, but hopefully it can be closed by improving the models, training algorithms etc.

Re: ChatGPT's Chess Elo is 1400

#177

Post by rzzzt »

orbital-decay wrote: post #28
mjr00 wrote: post #9

Earlier quoted context omitted.

The AI has simply, and correctly, identified that cheating is the best way to win at something.

It doesn't even know the rules, let alone cheat. It predicts the notation from the massive amount of games seen during training. Edit: although thinking of it, it probably anazyled a shitload of chess books too. It might have a lot of knowledge compressed into the internal representation. So yeah, maybe it knows rules in some form and even some heuristics, after all. It just doesn't understand the importance of makin…

If you have played with ye olde flip phone's T9 predictive feature as a child, trying to compose entire messages just by accepting the next word that comes to the phone's mind... that's ChatGPT, with the small difference of giving waaay better suggestions for the next word. But other than that, there is no understanding in the black box whatsoever.

Re: ChatGPT's Chess Elo is 1400

#180

Post by johnfn »

btilly wrote: post #113
johnfn wrote: post #50

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

Sorry, but not every game is unique. The following game has been played millions of times. 1. e4 e5 2. Bc4 Bc5 3. Qh5? Nf6?? 4. Qxf7++ The game Go has a claim to every game being unique. But not chess. And particularly not if both players follow a standard opening which there is a lot of theory about. Opening books often have lines 20+ moves deep that have been played many times. And grandmasters will play into these…

You seem to be refuting a specific point of my argument which has little bearing on the overall point I was making.

All games were provided in the article. None of them were 4 move checkmates; nearly every one is longer than 20 moves and some are 40 or longer. There is simply no possible way that ChatGPT is regurgitating the exact same 40-move-long game it's seen before. You can check a chess database if you'd like; virtually all games longer than 20 moves are unique.

Post reply on HN