Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

261–270 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#261

Post by pattrn »

whimsicalism wrote: post #246
throwwwaway69 wrote: post #240

Earlier quoted context omitted.

I'm not really sure what to say here. Both the parent commenter and the author of the article had issues with ChatGPT supplying illegal moves. Both methods resulted in this. It sort of doesn't matter how we're trying to establish that it's a 1400 level player, there's no defined correct way to do this. Regardless of method we've disproven it's a 1400 level player due to these illegal moves.

> Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Re: ChatGPT's Chess Elo is 1400

#262

Post by vidarh »

illiarian wrote: post #233
vidarh wrote: post #226

Earlier quoted context omitted.

Under FIDE rules it's first a forfeit after the second illegal move, so if anything it would seem that the interpretation used by the article author underestimates its ELO ranking.

Nope, still not even close to what the author claims. If I understand it correctly, it made illegal moves in 3 out of 19 games. That's probably a few orders of magnitude more illegal moves than even a 1400 ELO player would make of their entire lifetime.

The ranking takes into account wins and losses, not illegal moves, and so the fact that it plays in a way where a higher proportion of its losses is down to illegal moves than a human player is not relevant to its ranking. It may suggest that the ranking ought to take that into account, but that's a separate issue.

Re: ChatGPT's Chess Elo is 1400

#263

Post by liampulles »

With respect to the points here on how illegal moves invalidate ChatGPT as a valid Chess AI, consider the following:

AI.v1: Feed ChatCPT the move history and ask for the next move.

AI.v2: Wrap AI.v1 with additional logic to try and translate any invalid response as a similar/close valid move, else treat it as a resignation.

I would say AI.v2 is a "valid" Chess AI since it obeys the rules of chess. Also I think it is reasonable given how minimal this "wrapping" is to attribute the underlying ability to ChatGPT.

Re: ChatGPT's Chess Elo is 1400

#264

Post by z3c0 »

vidarh wrote: post #259
z3c0 wrote: post #227

Earlier quoted context omitted.

They are disproving an assertion. Demonstrating that an alternate approach implodes the assertion is a perfectly acceptable route, especially when the original approach was cherry-picking successes and throwing out failures. I wish I could just make bullshit moves and get a higher chess ranking. Sounds nice.

It was not throwing out failures. It was treating even the first illegal move as a forfeit something which is stricter than FIDE rules .

You can spin it that way if you want to, but the result is essentially guiding it through a brute force of the first successful playthrough it can muster.

Re: ChatGPT's Chess Elo is 1400

#265

Post by PaulHoule »

eddsh1994 wrote: post #242
mynameisvlad wrote: post #230

Earlier quoted context omitted.

> something a 1400 ranked player would never do The fact that rules and articles exist describing what to do if you or your opponent makes an illegal move indicates this is not the case. Humans are also... human. They make mistakes. It may not happen often at 1400, but to say that it'll never happen is preposterous.

I can’t remember the last time I played an illegal move tbf, and I’ve played 7 games of chess this morning already to give you an idea of total games played

I read an article about a pro player who castled twice in a game and my son hates castling so I make a point of castling twice as often as I can to tease him and attempting other illegal moves as a joke but he never ends the game because of it.

If I was playing that monstrosity though I would play something crazy that is far out of the opening book and count on it making an illegal move.

Re: ChatGPT's Chess Elo is 1400

#266

Post by mynameisvlad »

ipaddr wrote: post #258
mynameisvlad wrote: post #247

Earlier quoted context omitted.

You have never made an illegal move, ever? The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”. My entire point is that it happens. Not often, but also not “never”.

A broken clock is correct two times a day. But my broken clock isn't 1400 player although it might seem to be.

Does that somehow prove the assertion of "something a 1400 ranked player would never do"?

Because all I'm hearing is talk about ChatGPT's abilities as a reply to me calling out an extreme statement as being extreme. Something the parent comment even admitted as being overly black and white.

Re: ChatGPT's Chess Elo is 1400

#267

Post by pattrn »

mynameisvlad wrote: post #247
eddsh1994 wrote: post #242

Earlier quoted context omitted.

I can’t remember the last time I played an illegal move tbf, and I’ve played 7 games of chess this morning already to give you an idea of total games played

You have never made an illegal move, ever? The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”. My entire point is that it happens. Not often, but also not “never”.

This argument is pretty flimsy. ChatGPT makes illegal moves frequently. In all my years of playing competitive chess (from 1000 to 2200), I have never seen an illegal move. I'm sure it has happened to someone, but it's extremely rare. ChatGPT does it all the time. No one is arguing that humans never make illegal moves; they're arguing that ChatGPT makes illegal moves at a significantly higher rate than a 1400 player does (therefore ChatGPT does not have a 1400 rating).

Edit: Without reading everything again, I'll assume someone said "never." They're probably assuming the reader understands that "never" really means "with an infinitesimal probability," since we're talking about humans. If you're trying to argue that "some 1400 player has made an illegal move at some point," then I agree with that statement, and I also think it's irrelevant since the frequency of illegal moves made by ChatGPT compared to the frequency of illegal moves made by a 1400 rated player is many orders of magnitudes higher.

Re: ChatGPT's Chess Elo is 1400

#268

Post by bcrosby95 »

ipaddr wrote: post #258
mynameisvlad wrote: post #247

Earlier quoted context omitted.

You have never made an illegal move, ever? The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”. My entire point is that it happens. Not often, but also not “never”.

A broken clock is correct two times a day. But my broken clock isn't 1400 player although it might seem to be.

Prove to me your clock is broken, I think it's just telling the future.

Re: ChatGPT's Chess Elo is 1400

#269

Post by whimsicalism »

pattrn wrote: post #261
whimsicalism wrote: post #246

Earlier quoted context omitted.

> Regardless of method we've disproven it's a 1400 level player due to these illegal moves. Explain your thought process here further if you don't mind.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Think blindfolded 1400 players, which is what this effectively is, would make illegal moves.

But even if it doesn't play like human 1400 players, if it can get to a 1400 elo while resigning games it makes illegal moves on, that seems 1400 level to me. And i bet that some 1400s do occasionally make illegal moves (missing pins) while playing otb

Re: ChatGPT's Chess Elo is 1400

#270

Post by imtringued »

chessexpeort1 wrote: post #61
jonathanstrange wrote: post #7

Not to nitpick but I don't think it can be said to be able to play chess when it also makes illegal moves.

Humans make illegal moves too. So much that there's a rule for it-- "Penalty for Illegal Moves The penalty for the first illegal move is to add two minutes to the opponent's time. A second illegal move results in loss of the game.". https://new.uschess.org/sites/default/files/media/documents/...

How many people change the colour of the enemy queen and place new pieces on the board?
Post reply on HN