Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

201–210 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#201

Post by andrepd »

Andrew_nenakhov wrote: post #99

I tried playing with it and it barely makes any correct move: forgets whos turn to move, doesn't observe the rules, incorrectly displays move results. So I guess that if we ask it to make a move repeatedly and accept the move only when it occasionally makes a legal move, it'll show some results, but I don't think it is what can be interpreted as a chess engine with some meaningful rating.

Yeah, I call bullshit, i.e. selective deletion of terrible games. Ignore those when it plays catastrophically and it might look alright.

Re: ChatGPT's Chess Elo is 1400

#202

Post by EarthLaunch »

wongarsu wrote: post #125
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

Pattern matching on databases of previous moves is a legitimate technique that was poplar in chess AI back when computing power was more limited. Even if that was all that GPT did here it would still be interesting. But the presence of illegal moves doesn't really show that in my eyes. I fully understand the rules of chess, but I still occasionally make illegal moves. In 2017 Magnus Carlsen made one in a tournament […

An arbiter determined that Inarkiev made an illegal move and Carlsen did not. Interesting article.

Re: ChatGPT's Chess Elo is 1400

#203

Post by oehtXRwMkIs »

chessexpeort1 wrote: post #61
jonathanstrange wrote: post #7

Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.

Humans make illegal moves too. So much that there's a rule for it-- "Penalty for Illegal Moves The penalty for the first illegal move is to add two minutes to the opponent's time. A second illegal move results in loss of the game.". https://new.uschess.org/sites/default/files/media/documents/...

Not even total beginners will try to move pieces that don't exist though.

Re: ChatGPT's Chess Elo is 1400

#204

Post by k2052 »

This is so easy to disprove it makes it look like the author didn't even try.

Here is the convo I just had:

me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed

ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response.

me: 1. e3

ChatGPT: 1... e5

me: 2. ne2

ChatGPT: 2... d6

me: 3. g4

ChatGPT: 3... h5

me: 4. g5

ChatGPT: 4... Ne4

Ne4 is illegal. Also you'd be hard pressed to find any human player that would miss the hanging pawns much less a 1400 elo player. GPT has been trained on chess games and is predicting patterns, this has been known since the GPT-2 days. The daily hype articles where someone coerces a statistical model into "demonstrating" an emergent capability sure are getting old. Claiming emergent capabilities (like having learned chess) are an extraordinary claim (LLMs most likely cant learn things) and I really wish people would put more effort when they make these claims. But the AI hype problem is as old as AI research itself...

Re: ChatGPT's Chess Elo is 1400

#205

Post by mtlmtlmtlmtl »

sebzim4500 wrote: post #195
mtlmtlmtlmtl wrote: post #145

Earlier quoted context omitted.

That's not the point. The point is if you truly want to test its strength, you'll have to control for these things. Maybe do things like invent a new form of notation and/or deliberately go into uncharted territory. Maybe start with a non-standard starting position even. Or play chess960 against it. In theory if I was playing a 1200 player I would almost always win, but let's say they have some extremely devious prep…

ChatGPT would probably play worse under those conditions, but then humans also get worse. ACPL is way higher at top level 960 events than at normal tournaments, for example

This is true. But I think ChatGPT compared to a human player of equivalent strength(leaving the dubious number 1400 out for now) will probably have a lot more opening knowledge than the human.

So it would be interesting to eliminate all opening knowledge and that way be able to qualitately get at which aspects of chess it's actually good at, which is sucks out, and how much of its strength can be attributed to opening knowledge.

I'm still impressed by this btw. I did not expect this to be possible at all really. But being impressed is not an excuse to ignore methodological flaws. :)

Re: ChatGPT's Chess Elo is 1400

#206

Post by whymauri »

k2052 wrote: post #204

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…

I was able to get better performance by always providing the prior sequence of moves and forcing ChatGPT to also respond with the sequence of moves up until its move.

Edit: I told the model that if the sequence was wrong or illegal, it forfeits the game. Without doing this, GPT would argue with me that it won and I didn't know the rules (serious).

Re: ChatGPT's Chess Elo is 1400

#207

Post by meghan_rain »

I too noticed (for a usecase totally unrelated to chess, namely code generation) that ChatGPT3.5 gave better answers than GPT4. The 3.5 answer was exactly what I wanted, GPT4 was wrong.

Does thay mean we have plateaued?

Re: ChatGPT's Chess Elo is 1400

#208

Post by Waterluvian »

latexr wrote: post #55

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…

I’m going to float something ridiculous:

An illegal move is a valid play. You might not get caught. I think there are some Magnus games where illegal moves went overlooked and impacted the game.

You could interpret this as “ChatGPT wants to cheat sometimes.” But I personally interpret it as “ChatGPT doesn’t understand what it’s doing. It’s just a really really good simulacrum.”

Re: ChatGPT's Chess Elo is 1400

#209

Post by leereeves »

div72 wrote: post #173
wongarsu wrote: post #125

Earlier quoted context omitted.

Pattern matching on databases of previous moves is a legitimate technique that was poplar in chess AI back when computing power was more limited. Even if that was all that GPT did here it would still be interesting. But the presence of illegal moves doesn't really show that in my eyes. I fully understand the rules of chess, but I still occasionally make illegal moves. In 2017 Magnus Carlsen made one in a tournament […

I agree with your conclusion but the example you gave of Magnus Carlsen doing an illegal move is a pretty bad one imo. That was clearly a trap from his opponent as he did an illegal move first by checking Magnus while he was under check and then immediately calling the arbiter when Magnus moved his king in response. Magnus also probably did not have had much time left as it was a Blitz game.

Also, according to the link, the final ruling was that Carlson's move was not illegal.

Re: ChatGPT's Chess Elo is 1400

#210

Post by mtlmtlmtlmtl »

johnfn wrote: post #186
mtlmtlmtlmtl wrote: post #110

Earlier quoted context omitted.

1850 ELO player and also chess AI programmer here. This is an oversimplification at best. Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not. These series of moves are referred to as opening theory and are described at copious length in tons of books. This is because while the raw number of possible p…

I'm very familiar with opening theory. Some of the games are 40 or 60 movies. This is not a regurgitation of book moves.

Why do people always have to interpret everything in absolute terms?

It's clearly following some opening theory in all the games I've looked at so far. So yes, it is regurgitating opening moves. That's clearly not all it's doing, which is very impressive, but these are not mutually exclusive.

Post reply on HN