Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

151–160 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#151

Post by jmull »

vidarh wrote: post #95
latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

The article also says in one game chatgpt when crazy so they continued the game with a fresh chat. That probably should have been counted a resignation loss too.

Re: ChatGPT's Chess Elo is 1400

#152

Post by macNchz »

yowzadave wrote: post #57
PoignardAzur wrote: post #35

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?

Doesn't ChatGPT indeed have a very high number of illegal moves? https://www.youtube.com/watch?v=kvTs_nbc8Eg In this example, ChatGPT's first few moves are reasonable (while it appears to be on-book), but then it goes off the rails and starts moving illegally, spawning pieces out of nowhere, deleting pieces for no reason, etc.

Reminds me of asking for driving directions (city to city)...for major cities it can often give perfect directions, for smaller cities it starts out surprisingly accurate but often devolves into invented exits or descriptions of a

Re: ChatGPT's Chess Elo is 1400

#153

Post by swatcoder »

sebzim4500 wrote: post #108
Out_of_Characte wrote: post #100

Earlier quoted context omitted.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.

ChatGPT did forfeit whenever it made an illegal move, read the article.

No, the writer arbitrarily decided to interpret illegal moves as resignations in order to support the conclusion they wanted. That's very different and grossly unscientific.

Re: ChatGPT's Chess Elo is 1400

#154

Post by dragontamer »

oezi wrote: post #116
dragontamer wrote: post #104

Earlier quoted context omitted.

I'm Elo 1400 and can beat literally everyone I know in the real world. I need to go online to find players at my skill level, or find tournament/competitive settings for a challenge. Yeah, I'm "class C", weak amateur chess player, but I think you're grossly underestimating the amount of study I put into this game. I'm not going to make an illegal move

I mean can you play just based on being provided the input of a series of moves without it being shown to you as a visual board? I guess most players would mess up 20/30 moves in.

https://chessily.com/blog/stockfish-vs-chatgpt/

> ChatGPT: Yes, that’s a good move for you. My next move is: Bc3, developing my pieces and attacking your pawn on c3.

I am 1400 Elo and can tell you that from an near opening position, its impossible to move a Bishop to c3 for either Black or White in the first say, 10 moves, under traditional openings.

Re: ChatGPT's Chess Elo is 1400

#155

Post by sebzim4500 »

LeanderK wrote: post #128
marcodiego wrote: post #79

This may look low: ELO for mediocre players is 1500. But if it is obeying the rules of the game, then this is big. This is a signal that if it learns some expertise, like discovering how to use or create better search algorithms (like MCTS and heuristics to evaluate a state) and improve by itself (somewhat like alphazero did), then it may eventually reach superhuman level. It may then reach superhuman level in any ta…

I think current LLM architectures are limiting the strategies it learned. MCTS requires recursion but GPT is always executing a fixed number of steps. Allowing language models more flexibility by a variable number of steps, for example through recursion of the model, would eliminate this hard bound but they are harder to design and train. We have just been able to train GPT-sized models.

I'm sure the MuZero chess policy network would reach much higher than 1400, and that has no notion of recursion either. (And also wasn't taught the rules explicitly)

Re: ChatGPT's Chess Elo is 1400

#156

Post by medvezhenok »

dragontamer wrote: post #142
medvezhenok wrote: post #126

Earlier quoted context omitted.

You will under time pressure :) even Grandmasters have done that (I'm around 1900 elo for context) Also people forgetting they moved the king/rook and trying to castle.

Watch this ChatGPT game. https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... We're talking about pieces that don't exist, reappearing pieces, pieces moving completely wrong (Knight takes as if its a Pawn), etc. etc. --------- People are taking these example games and saying ChatGPT is 1400 strength. I don't think so. This isn't a case of "oops, I castled even though I moved my king 15 turns ago".

The article points out that the way that game was conducted was bad. (Here's the original transcript: https://pastebin.com/X6kBRTa9)

You need to give ChatGPT the full state (every move) on every prompt to make it play closer to 1400. The game you linked the user was giving one move at a time.

Re: ChatGPT's Chess Elo is 1400

#157

Post by saghm »

vidarh wrote: post #95
latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

Fuller context from the article: > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. (my emphasis) So the illegal moves are at least part of the reasons for the 6 losses, and factored into…

So it sounds like it can play _some_ legal chess games, but not all; it's unable to consistently complete a game where it loses. Maybe the remaining work shouldn't be focused on trying to teach it chess rules better, but to teach it sportsmanship better. People were so excited about teaching it high-school level academics that we forgot to teach it the basic lessons we learn in kindergarten.

Re: ChatGPT's Chess Elo is 1400

#158

Post by dragontamer »

medvezhenok wrote: post #156
dragontamer wrote: post #142

Earlier quoted context omitted.

Watch this ChatGPT game. https://www.reddit.com/r/AnarchyChess/comments/10ydnbb/i_pla... We're talking about pieces that don't exist, reappearing pieces, pieces moving completely wrong (Knight takes as if its a Pawn), etc. etc. --------- People are taking these example games and saying ChatGPT is 1400 strength. I don't think so. This isn't a case of "oops, I castled even though I moved my king 15 turns ago".

The article points out that the way that game was conducted was bad. (Here's the original transcript: https://pastebin.com/X6kBRTa9 ) You need to give ChatGPT the full state (every move) on every prompt to make it play closer to 1400. The game you linked the user was giving one move at a time.

> You need to give ChatGPT the full state

If I've been given the full state every move, I will _never_ make an illegal move as a 1400 chess player.

-----------

> O-O > I'll play O-O as well. Your move.

Do you really think that this error would have been made at 1400 Elo? Even in blind chess? This is the 5th move of the game. I can still track the game at this point mentally.

I recognize that you're 1900 and think that all the chess players below you are n00bs, but... come on. 1400 players are stronger than this.

Re: ChatGPT's Chess Elo is 1400

#159

Post by sebzim4500 »

qwytw wrote: post #133
dwmbt wrote: post #17

Earlier quoted context omitted.

slight nit: this is true amongst all chess players. amongst professionals and amateurs, 1400 is technically Class C - which is on the lower end of amateurs. nonetheless, it's cool to see a model 'generally intelligent' enough to beat most players. what's more interesting to me is that according to OP's analysis, GPT4 is worse at Chess than GPT3.5... source: https://en.wikipedia.org/wiki/Chess_rating_system

It would depend how often did this happen: > Occasionally it does make an illegal move Since it would lose every game in which it made more than a single illegal move.

Well it can't have happened more than 6 times since illegal moves counted as a forfeit and it only lost 6 games.
Post reply on HN