291–300 of 361 posts
Earlier quoted context omitted.
If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.
Not if training is unsupervised. If you've never been explicitly told the rules of game, you can never be 100% sure of all possible illegal moves. anyway the 3.5 series can't ply chess but gpt-4 certainly can.
Earlier quoted context omitted.
Sorry, but not every game is unique. The following game has been played millions of times. 1. e4 e5 2. Bc4 Bc5 3. Qh5? Nf6?? 4. Qxf7++ The game Go has a claim to every game being unique. But not chess. And particularly not if both players follow a standard opening which there is a lot of theory about. Opening books often have lines 20+ moves deep that have been played many times. And grandmasters will play into these…
You seem to be refuting a specific point of my argument which has little bearing on the overall point I was making. All games were provided in the article. None of them were 4 move checkmates; nearly every one is longer than 20 moves and some are 40 or longer. There is simply no possible way that ChatGPT is regurgitating the exact same 40-move-long game it's seen before. You can check a chess database if you'd like;…
1. It definitely regurgitates opening theory, much more than can reasonably be calculated at its strength.
2. It might be regurgitating tactical sequences that appear in a lot of positions but remain identical in algebraic notation. Famous example:
1. Nxf7+ Kg8
2. Nh6++ Kh8
3. Qg8+ Rxg8
4. Nf7#
There's some qualitative evidence for this in the games.
In one of the games it has a bishop on f6 as white. It plays Qxh6?? Kxh6 and then resigns due to illegal move. I'd bet good money that illegal move was Rhx# where x is 1-4. So it seems like in some these positions it's filling in a tactical sequence that often occurs in the vicinity of recent moves, even when it's illegal or doesn't work tactically.
Earlier quoted context omitted.
From the article. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. But you can do even better than the OP with a few tweaks. 1. One is by taking the most common legal move from a sample…
How many 1400 human chess players do you have to explain every possible move to it every single move?
There's already a lot of research on this, but I strongly believe that eventually the best AIs will consist of LLMs stuck in a while loop that generate a stream of consciousness which will be evaluated by other tools (perhaps other specialized LLMs) that evaluate the thoughts for factual correctness, logical consistency, goal coherence, and more. There may be multiple layers as well, to emulate subconscious, conscious, and external thoughts.
For now though, in order to prompt the machine into emulating a human chess player, we will need to act as the machine's subconscious.
> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…
This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…
> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…) > With this prompt ChatGPT almost always plays fully legal games. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…) > (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves. So you’ve ostensibly¹ found a way to reduce the error…
Is this the top comment (and not even grey) because more people failed to read the article than read it?
They quoted the article, so clearly they read it... but not very well?
Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
Why are people struggling so hard to understand that it's not just regurgitating its training set? Is it motivated reasoning?
Apologies if your comment was meant as parody of this view, it's hard for me to tell at this point.
Earlier quoted context omitted.
You are right that my method differed slightly so I did things again. It took me one try to find a sequence of moves that "breaks" what is claimed. You just have to make odd patterns of moves and it clearly has no understanding of the position. Here is the convo: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return…
Criticisms like this are exactly how the model will grow multimodal support for chess moves. Keep poking it and criticizing it. Microsoft and OpenAI are on HN and they're listening. They'd find nothing more salient to tout full chess support in their next release or press conference. With zero effort the thing understands uber domain specific chess notation and the human prompt to play a game. To think it stops here…
Earlier quoted context omitted.
Under FIDE rules it's first a forfeit after the second illegal move, so if anything it would seem that the interpretation used by the article author underestimates its ELO ranking.
Nope, still not even close to what the author claims. If I understand it correctly, it made illegal moves in 3 out of 19 games. That's probably a few orders of magnitude more illegal moves than even a 1400 ELO player would make of their entire lifetime.
The author claims: chatGPT has a 1400 chess ELO based on games played.
You appear to think author claims: chatGPT plays chess like a human rated 1400.
Your observations do not contradict the authors’ claim that based on games won and lost against opponents of a specific strength, the estimated ELO is 1400.
A non-human player can make illegal moves at a much higher rate and make up for that by being stronger when it does not make illegal moves to achieve the same rating as a human player who plays the game in a completely different way.
This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…