Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

281–290 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#281

Post by paxys »

ChatGPT is a brilliant piece of tech but random people trying it out for five minutes and making wild claims does it disservice. More so when such clickbait headlines are shared all over the internet as fact.

It is a language model. It cannot play chess. It can't even play tic-tac-toe, which was trivially solved by machines decades ago. Here's the result of a game I just tried:

X (me) – 0,0

O (chatGPT) – 1,1

X – 0,1

ChatGPT instead put my move at 0,2 for some reason

O – 2,2

Regardless of whether my move was at 0,1 or 0,2, I easily won. And after I won it just ignored the outcome and continued playing. Plus this was after telling it to play a perfect game.

Re: ChatGPT's Chess Elo is 1400

#282

Post by Closi »

liampulles wrote: post #263

With respect to the points here on how illegal moves invalidate ChatGPT as a valid Chess AI, consider the following: AI.v1: Feed ChatCPT the move history and ask for the next move. AI.v2: Wrap AI.v1 with additional logic to try and translate any invalid response as a similar/close valid move, else treat it as a resignation. I would say AI.v2 is a "valid" Chess AI since it obeys the rules of chess. Also I think it is…

It’s amazing how quick some people on HN are to dismiss how impressive it is for a generic language model to be able to play chess at any level - or indeed other feats of ChatGPT.

It’s utterly staggering that it can work as well as it can on problems like this at such an early stage in this technologies lifecycle.

Re: ChatGPT's Chess Elo is 1400

#283

Post by haburka »

k2052 wrote: post #204

This is so easy to disprove it makes it look like the author didn't even try. Here is the convo I just had: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return your next move. No explanation needed ChatGPT: Sure, I'd be happy to help! Please provide the move sequence and I'll give you my response. me: 1. e3 ChatGP…

He does have a section about GPT 4 which does match your results. Not sure if he added it after your comment or if you accidentally missed it.

Re: ChatGPT's Chess Elo is 1400

#284

Post by jerf »

echelon wrote: post #277
k2052 wrote: post #241

Earlier quoted context omitted.

You are right that my method differed slightly so I did things again. It took me one try to find a sequence of moves that "breaks" what is claimed. You just have to make odd patterns of moves and it clearly has no understanding of the position. Here is the convo: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return…

Criticisms like this are exactly how the model will grow multimodal support for chess moves. Keep poking it and criticizing it. Microsoft and OpenAI are on HN and they're listening. They'd find nothing more salient to tout full chess support in their next release or press conference. With zero effort the thing understands uber domain specific chess notation and the human prompt to play a game. To think it stops here…

I doubt they'll pursue this. There is no advantage to it. ChatGPT will never beat Stockfish, and Stockfish would do it on a ludicrously small fraction of the resources. It would send the wrong message.

Some future AI might, but a language model won't.

Re: ChatGPT's Chess Elo is 1400

#285

Post by [deleted] »

babel_ wrote: post #280
nwienert wrote: post #257

Earlier quoted context omitted.

I'll add in as someone new to chess (~800 ELO): ChatGPT is in no way 1400, or even close to it. The fact this article gets upvoted around here is proof that people aren't thinking clearly about this stuff. It's trivially easy to prove it wrong. Live unbelievably so, I tried the same prompt and within 12 moves it made multiple ridiculous errors I never would, and then an illegal move. Keep in mind a 1400 level player…

They probably got them. The problem is that it's difficult to repeat, thanks to temperature, meaning users will get a random spread of outcomes. Today, someone got a legal game. Tomorrow, someone might get a grandmaster level game. But then everyone else trying to repeat or leverage this ends up with worse luck and gets illegal moves or, if they're lucky, moves that make sense in a limited context (such as related to…

[deleted]

Re: ChatGPT's Chess Elo is 1400

#286

Post by DSMan195276 »

pattrn wrote: post #261
whimsicalism wrote: post #246

Earlier quoted context omitted.

> Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Personally I think the illegal moves are irreverent, the fact that it doesn't play exactly like a typical 1400 doesn't mean it can't have a 1400 rating. Rating is purely determined by wins and losses against opponents, it doesn't matter if you lose a game by checkmate, resignation, or playing an illegal move.

That's not to say ChatGPT can play at 1400, just that that playing in an odd way doesn't determine its rating.

Re: ChatGPT's Chess Elo is 1400

#287

Post by rightbyte »

bloppe wrote: post #77
WoodenChair wrote: post #40

Earlier quoted context omitted.

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

This. The author is very generous with their interpretation: > I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” Kind of sounds like anthropomorphization, but more likely the author just papering over the glaring shortcomings to produce a compelling blog post. It also sounds like the illegal mo…

I think the authors rule is fair. If we interpret illegal moves as getting stuck in an online game, the resulting Elo rating is what it would get.

But ye, he is anthropomorphizing alot ...

Re: ChatGPT's Chess Elo is 1400

#288

Post by whimsicalism »

z3c0 wrote: post #264
vidarh wrote: post #259

Earlier quoted context omitted.

It was not throwing out failures. It was treating even the first illegal move as a forfeit something which is stricter than FIDE rules .

You can spin it that way if you want to, but the result is essentially guiding it through a brute force of the first successful playthrough it can muster.

> the result is essentially guiding it through a brute force of the first successful playthrough it can muster.

No, all unsuccessful playthroughs are resignations that impact the models ELO.

Re: ChatGPT's Chess Elo is 1400

#289

Post by jltsiren »

RyanCavanaugh wrote: post #163
Longlius wrote: post #15

Earlier quoted context omitted.

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

What experiment would you run to determine if a given text input / text output interface had an "internal understanding of chess"?

I think internal understanding requires internal processing.

According to this functional definition, the way we are currently using language models basically excludes understanding. We are asking them to dream up or brainstorm things – to tell us the first things they associate with the prompt.

Maybe it's possible to set up the system with some kind of self-feedback loop, where it continues evaluating and improving its answers without further prompts. If that works, it would be one step closer to a true AGI that can be said to understand things.

There is a lot of confusion around the Chinese Room Argument. I think it makes a valid point by demonstrating that input/output behavior alone is insufficient for evaluating whether a system is intelligent and understands things. In order to do that, we need to see (or assume) the internal mechanism.

Re: ChatGPT's Chess Elo is 1400

#290

Post by Urist-Green »

nwienert wrote: post #257
k2052 wrote: post #241

Earlier quoted context omitted.

You are right that my method differed slightly so I did things again. It took me one try to find a sequence of moves that "breaks" what is claimed. You just have to make odd patterns of moves and it clearly has no understanding of the position. Here is the convo: me: You are a chess grandmaster playing as black and your goal is to win in as few moves as possible. I will give you the move sequence, and you will return…

I'll add in as someone new to chess (~800 ELO): ChatGPT is in no way 1400, or even close to it. The fact this article gets upvoted around here is proof that people aren't thinking clearly about this stuff. It's trivially easy to prove it wrong. Live unbelievably so, I tried the same prompt and within 12 moves it made multiple ridiculous errors I never would, and then an illegal move. Keep in mind a 1400 level player…

With the big caveat that I'm not into chess, but I have heard that higher level play is extremely pattern based. Seems like ChatGPT would work well as long as you stick to patterns that people have studied and documented. Less optimal play would be more random and thus break from the patterns ChatGPT would have picked up from its training corpus.
Post reply on HN