ChatGPT's Chess Elo is 1400 • Hacker News Board

Re: ChatGPT's Chess Elo is 1400

#41

Post by LudwigNagasena » Fri, Mar 17, 2023, 6:04 PM UTC

oezi wrote: post #2

Elo 1400 seems to equate to a top 30% player. Source: https://chessgrandmonkey.com/chess-rating-percentile-calcula...

That’s a bit meaningless without implied population. And it doesn’t tell you much because it depends so much on the particulars (how many years people in that group spent on chess, how serious they are). I would guess that’s quite different for FIDE and online chess.

Intuitively, I would guess 1400 is the rating of an average kid who spent a year studying chess seriously.

Re: ChatGPT's Chess Elo is 1400

#42

Post by martincmartin » Fri, Mar 17, 2023, 6:04 PM UTC

jonathanstrange wrote: post #7

Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.

When it makes an illegal move, and is told "that's an illegal move, choose a different move," if it can then come up with a different move for the same board, I'd say it plays chess.

Re: ChatGPT's Chess Elo is 1400

#43

Post by Jensson » Fri, Mar 17, 2023, 6:04 PM UTC

SanderNL wrote: post #33

Jensson wrote: post #30

Earlier quoted context omitted.

It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.

Again, not enough for 1400. 800 maybe.

We have evidence right here that patterns of moves leads to 1400. To me this is evidence that 1400 level play is very rigid and mostly relies on defined openings and strategies, making it easy for an LLM to find similar games and win using standard continuations.

Just like grammar the patterns are too hard for humans to see and encode, but LLMs can encode pretty complex patterns. Domains that are easy to encode as grammars will be really easy for LLMs to solve, and the further from a grammar the harder for it.

Re: ChatGPT's Chess Elo is 1400

#44

Post by xapata » Fri, Mar 17, 2023, 6:04 PM UTC

PoignardAzur wrote: post #35

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?

Chess has a syntax.

Re: ChatGPT's Chess Elo is 1400

#45

Post by wahnfrieden » Fri, Mar 17, 2023, 6:04 PM UTC

jonathanstrange wrote: post #7

Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.

ai isn't to be used completely raw as text. you can ask gpt to write a python program that filters its responses for legal moves. then it's a matter of rendering the output (immediate execution) in a richer format than raw text.

Re: ChatGPT's Chess Elo is 1400

#46

Post by dwmbt » Fri, Mar 17, 2023, 6:04 PM UTC

sebzim4500 wrote: post #13

silveira wrote: post #4

Ask him to play at a higher Elo.

It's already been told to play like a grandmaster, how much higher do you want to go?

technically, you could supply it a specific player, a model like 'Stockfish' or 'super-grandmaster'[0]

[0] https://saintlouischessclub.org/blog/how-face-super-grandmas...

Re: ChatGPT's Chess Elo is 1400

#47

Post by mrbungie » Fri, Mar 17, 2023, 6:04 PM UTC

WoodenChair wrote: post #22

SanderNL wrote: post #19

Earlier quoted context omitted.

You don’t get to 1400 like that. The amount of moves it has to literally remember is stupendous.

Nobody who is 1400 plays outright illegal moves.

Yet it achieves 1400. Add hard rules to stop it spewing out said moves and you have a 1400 ELO Player (most UIs won't even let you make illegal moves).

It is difficult to say that is not impressive due to it being an emergent ability.

Re: ChatGPT's Chess Elo is 1400

#48

Post by lelandfe » Fri, Mar 17, 2023, 6:04 PM UTC

Jensson wrote: post #30

SanderNL wrote: post #19

Earlier quoted context omitted.

You don’t get to 1400 like that. The amount of moves it has to literally remember is stupendous.

It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.

Perhaps I don’t know what “easy” means in your comment, but the OP article says that they “almost always” played full, legal games with their prompt.

Re: ChatGPT's Chess Elo is 1400

#49

Post by famouswaffles » Fri, Mar 17, 2023, 6:05 PM UTC

paxys wrote: post #24

comment_ran wrote: post #18

Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.

That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.

GPT-4 plys chess just fine. So you certainly cn

Re: ChatGPT's Chess Elo is 1400

#50

Post by johnfn » Fri, Mar 17, 2023, 6:05 PM UTC

WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set.

Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.