ChatGPT's Chess Elo is 1400

Post by illiarian » Fri, Mar 17, 2023, 7:27 PM UTC

wongarsu wrote: post #125

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

Pattern matching on databases of previous moves is a legitimate technique that was poplar in chess AI back when computing power was more limited. Even if that was all that GPT did here it would still be interesting. But the presence of illegal moves doesn't really show that in my eyes. I fully understand the rules of chess, but I still occasionally make illegal moves. In 2017 Magnus Carlsen made one in a tournament […

Post by ogogmad » Fri, Mar 17, 2023, 7:28 PM UTC

Jensson wrote: post #218

jart wrote: post #196

Earlier quoted context omitted.

I personally find that makes it more astonishing, that it would slip up on knowing the most basic elements of the game, yet still be able to play better than most humans. Highly smart people sometimes say or do little things when foraying into other fields that causes domain experts think they're not one of them. But that usually doesn't stop smart people from having an impact in making a contribution with their insi…

> Highly smart people sometimes say or do little things when foraying into other fields that causes domain experts think they're not one of them This is the opposite of that, a highly trained but dumb entity that has seen many lifetimes worth of games but is still tripping up on basics. But since it is so highly trained you can mistake it for a master if you squint and don't look into what it is doing.

Post by ogogmad » Fri, Mar 17, 2023, 7:30 PM UTC

hectorlorenzo wrote: post #147

dudeinjapan wrote: post #91

Earlier quoted context omitted.

Obviously the article should be taken with a giant grain of salt. That being said, not many things what aren't designed to play chess can play chess, with or without coaxing. My dog cannot, for instance, nor can my coffee table.

> My dog cannot, for instance, nor can my coffee table. You must be giving them the wrong prompts.

Post by z3c0 » Fri, Mar 17, 2023, 7:31 PM UTC

vidarh wrote: post #212

k2052 wrote: post #204

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…

You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as well as the article claims at least use the same method.

Post by Jensson » Fri, Mar 17, 2023, 7:34 PM UTC

ogogmad wrote: post #223

Jensson wrote: post #218

Earlier quoted context omitted.

> Highly smart people sometimes say or do little things when foraying into other fields that causes domain experts think they're not one of them This is the opposite of that, a highly trained but dumb entity that has seen many lifetimes worth of games but is still tripping up on basics. But since it is so highly trained you can mistake it for a master if you squint and don't look into what it is doing.

> But since it is so highly trained you can mistake it for a master if you squint and don't look into what it is doing. But it is a master, as has been pointed out repeatedly. If you replace all illegal moves with resignations, and use the same style of prompt as the OP did, then it plays like an expert. I'm objecting because you're making it sound like it's a trivial result.