Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

301–310 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#301

Post by nottathrowaway3 »

k2052 wrote: post #204

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…

> me: You are a chess grandmaster playing as black...

https://upload.wikimedia.org/wikipedia/en/5/5f/Ingmar_Bergma...

The KNIGHT holds out his two fists to CHATGPT, who smiles at him suddenly. CHATGPT points to one of the KNIGHT'S hands; it contains a black pawn.

KNIGHT: You drew black.

CHATGPT: Very appropriate. Don't you think so?

Re: ChatGPT's Chess Elo is 1400

#302

Post by charcircuit »

swatcoder wrote: post #144
vidarh wrote: post #95

Earlier quoted context omitted.

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

No ELO 1400 player will have that rate of illegal moves, so saying it that it plays with an ELO 1400 rating is disingenuous. Reinterpreting illegal moves as resignation is absurd when an LLM is formally capable of expressing statements "I resign" or "I cannot conceive of a winning move from here" just as well as any human player. It just doesn't do so because it's not actually playing chess the way we think of an ELO…

ELO is based off who you win and lose against. The rate of illegal moves has nothing to do with ELO.

Re: ChatGPT's Chess Elo is 1400

#304

Post by manquer »

CSMastermind wrote: post #184
mtlmtlmtlmtl wrote: post #175

There's a huge difference between 1400 elo in FIDE games versus 1400 on chess.com, which is not even using elo. For instance the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating. Chess.com and lichess have a ton of rating inflation.

> the strongest blitz players in the world are hundreds of points higher rated on chess.com blitz versus their FIDE blitz rating Online rating inflation is real but I'm not sure blitz is the best example of it because in that case there is a notable difference between online and otb (having to take time to physically move the pieces).

Probably the bigger difference is ability to premove online

Re: ChatGPT's Chess Elo is 1400

#305

Post by [deleted] »

the_af wrote: post #148

This is GPT4, right? Because ChatGPT (GPT-3) still fails to provide a legal game of Tic Tac Toe with this prompt: > "Let's play Tic Tac Toe. You are O, I'm X. Display the board in a frame, with references for the axes" It failed to recognize that I won. Then continued playing (past the end), played illegally over a move I had already done, obtained a line of 3 for itself, and still doesn't acknowledge the game has en…

[deleted]

Re: ChatGPT's Chess Elo is 1400

#307

Post by ptmx »

mtlmtlmtlmtl wrote: post #245
ptmx wrote: post #118

Earlier quoted context omitted.

I played chess against ChatGPT4 a few days ago without any special prompt engineering, and it played at what I would estimate to be a ~1500-1700 level without making any illegal moves in a 49 move game. Up to 10 or 15 moves, sure, we're well within common openings that could be regurgitated. By the time we're at move 20+, and especially 30+ and 40+, these are completely unique positions that haven't ever been reached…

I'd be interested in seeing this game, if you saved it?

I uploaded the PGN to lichess: https://lichess.org/rzSriO6I#97

After reviewing the chat history I actually have to issue a correction here, because there were two moves where ChatGPT played illegally:

1. ChatGPT tried to play 32. ... Nc5, despite there being a pawn on c5

2. ChatGPT tried to play 42. ... Kxe6, despite my king being on d5

It corrected itself after I questioned whether the previous move was legal.

I was pretty floored that it managed to play a coherent game at all, so evidently I forgot about the few missteps it made. Much like ChatGPT itself, it turns out I'm not an entirely reliable narrator!

Re: ChatGPT's Chess Elo is 1400

#308

Post by [deleted] »

DSMan195276 wrote: post #286
pattrn wrote: post #261

Earlier quoted context omitted.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Personally I think the illegal moves are irreverent, the fact that it doesn't play exactly like a typical 1400 doesn't mean it can't have a 1400 rating. Rating is purely determined by wins and losses against opponents, it doesn't matter if you lose a game by checkmate, resignation, or playing an illegal move. That's not to say ChatGPT can play at 1400, just that that playing in an odd way doesn't determine its rating…

[deleted]

Re: ChatGPT's Chess Elo is 1400

#309

Post by bloppe »

rightbyte wrote: post #287
bloppe wrote: post #77

Earlier quoted context omitted.

This. The author is very generous with their interpretation: > I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” Kind of sounds like anthropomorphization, but more likely the author just papering over the glaring shortcomings to produce a compelling blog post. It also sounds like the illegal mo…

I think the authors rule is fair. If we interpret illegal moves as getting stuck in an online game, the resulting Elo rating is what it would get. But ye, he is anthropomorphizing alot ...

There's no indication that GPT-3.5 was stuck when it tried to make illegal moves. GPT-4 clearly was making illegal moves when it was very much not stuck. It just doesn't know how to play, but the author decided to interpret it as frustration.

Re: ChatGPT's Chess Elo is 1400

#310

Post by johnfn »

mtlmtlmtlmtl wrote: post #210
johnfn wrote: post #186

Earlier quoted context omitted.

I'm very familiar with opening theory. Some of the games are 40 or 60 movies. This is not a regurgitation of book moves.

Why do people always have to interpret everything in absolute terms? It's clearly following some opening theory in all the games I've looked at so far. So yes, it is regurgitating opening moves. That's clearly not all it's doing, which is very impressive, but these are not mutually exclusive.

I am responding to OP, who said "Most likely it has seen a similar sequence of moves in its training set."

From this, I take it that the question is if ChatGPT is repeating existing games, or not. All you need is a single game where it's not repeating a single game to prove it definitively. You can hardly play 60 moves without an error by accident.

I believe you're responding to a different question, something like "does ChatGPT fully understand the game of chess".

Post reply on HN