Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

51–60 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#51

Post by TOMDM »

RC_ITR wrote: post #34
comment_ran wrote: post #18

Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.

This is actually a huge debate right now. OpenAI is on the side of 'LLMs have only surprised us to the upside, so using crutches is counterproductive' Whereas other people think 'Teaching an LLM to do arbitrary math problems through brute force is probably one of the most wasteful things imaginable when calculators exist.' I'm actually very excited to see which side wins (I'm on team calculator, but want to be on tea…

I think I'm on both teams.

As GPT models get better at arbitrary computation, it stands to reason that their ability to use tools will rise as well.

Smarter models and smarter ways to get them using tools feel like complementary capabilities.

Re: ChatGPT's Chess Elo is 1400

#52

Post by [deleted] »

porphyra wrote: post #39
paxys wrote: post #24

Earlier quoted context omitted.

That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.

What if you come up with a prompt to tell ChatGPT to figure out how to use APIs and external tools to help itself get better at chess?

[deleted]

Re: ChatGPT's Chess Elo is 1400

#53

Post by nonethewiser »

sebzim4500 wrote: post #13
silveira wrote: post #4

Ask him to play at a higher Elo.

It's already been told to play like a grandmaster, how much higher do you want to go?

It should be compared to a simpler prompt with less instruction ("what move should be played in this chess game: ..." or something). I've found that additional constraints sometimes aren't applied very well and actually result in worse output.

Re: ChatGPT's Chess Elo is 1400

#54

Post by henryfjordan »

sebzim4500 wrote: post #20
Longlius wrote: post #15

Earlier quoted context omitted.

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

Mostly it didn't make illegal moves though, since illegal moves mean resignation and it won more than it lost. Making 60 legal moves in a row in one game would be the coincidence of the century unless it had some knowledge of the rules of chess.

It's a probabilistic text model. If it has a 99% probability of generating an acceptable "next" thing to say, that means it would have a 50/50 chance of generating 60 legal moves in a row, which doesn't seem all that coincidental.

Re: ChatGPT's Chess Elo is 1400

#55

Post by latexr »

> These people used bad prompts and came to the conclusion that ChatGPT can’t play a legal chess game. (…)

> With this prompt ChatGPT almost always plays fully legal games.

> Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table (…)

> (…) with GPT4 (…) in the two games I attempted, it made numerous illegal moves.

So you’ve ostensibly¹ found a way to reduce the error rate and then deliberately ignored the cases where it failed. In short: it may play valid chess under certain conditions but can’t be trusted to do so. That doesn’t contradict previous findings.

¹ 19 games is a small sample and the supposedly more advanced system failed in your tries.

Re: ChatGPT's Chess Elo is 1400

#56

Post by root_axis »

chpatrick wrote: post #21
Longlius wrote: post #15

Earlier quoted context omitted.

It is incapable of reasoning, actually - at least in this case. It has no internal understanding of chess which is why it makes illegal moves.

How did it win 11 out of 19 games then, blind luck?

raw statistical power.

Re: ChatGPT's Chess Elo is 1400

#57

Post by yowzadave »

PoignardAzur wrote: post #35
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?

Doesn't ChatGPT indeed have a very high number of illegal moves?

https://www.youtube.com/watch?v=kvTs_nbc8Eg

In this example, ChatGPT's first few moves are reasonable (while it appears to be on-book), but then it goes off the rails and starts moving illegally, spawning pieces out of nowhere, deleting pieces for no reason, etc.

Re: ChatGPT's Chess Elo is 1400

#58

Post by stagger87 »

paxys wrote: post #24
comment_ran wrote: post #18

Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.

That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.

Honest questions,

Isn't it very likely chess games were part of the training data? If so, chess sequences seem like an amazing matched filter of sorts. It doesn't seem surprising that it can guess a next move.

Is it teaching itself anything here? Is the model being updated in any way? Or are you talking more generally, looking into the future sort of statement?

Re: ChatGPT's Chess Elo is 1400

#59

Post by xapata »

LudwigNagasena wrote: post #41
oezi wrote: post #2

Elo 1400 seems to equate to a top 30% player. Source: https://chessgrandmonkey.com/chess-rating-percentile-calcula...

That’s a bit meaningless without implied population. And it doesn’t tell you much because it depends so much on the particulars (how many years people in that group spent on chess, how serious they are). I would guess that’s quite different for FIDE and online chess. Intuitively, I would guess 1400 is the rating of an average kid who spent a year studying chess seriously.

Not the ELO of the average kid who spends a year studying chess seriously? It's very difficult to define the average kid without context.

It's been a while since I read a global demographics update, but not all that long ago the mode child would have been living on $2 a day.

Re: ChatGPT's Chess Elo is 1400

#60

Post by scarmig »

WoodenChair wrote: post #40
PoignardAzur wrote: post #35

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

We would expect some vanishingly small fraction. Even at the grandmaster level, they happen (rarely):

https://chess24.com/en/read/news/the-7-most-illegal-chess-mo...

Post reply on HN