Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

321–330 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#321

Post by teej »

dsjoerg wrote: post #314
teej wrote: post #303

I just deployed a GPT-4 powered chess bot to lichess. You can challenge it here: https://lichess.org/@/oopsallbots-gpt-4

Very cool! Are you doing prompt engineering, fine-tuning, both, something else? I'm wondering if it'd be cool to have a chess contest where all the bots are LLM powered. Seems to me like the contest would have to ban prompt engineering -- would have to have a fixed prompt -- otherwise people would sneak chess engines into their prompt generation.

I wanted it to be fun and actually complete games. I started with this, with a few minor tweaks: https://github.com/Tmate6/Lichess_ChatGPT_ChatBot/blob/main/...

This approach sends along the list of legal moves in the prompt if it attempts an illegal move. That seems to work well at getting playable moves.

Re: ChatGPT's Chess Elo is 1400

#322

Post by eddsh1994 »

mynameisvlad wrote: post #247
eddsh1994 wrote: post #242

Earlier quoted context omitted.

I can’t remember the last time I played an illegal move tbf, and I’ve played 7 games of chess this morning already to give you an idea of total games played

You have never made an illegal move, ever? The bar isn’t “I didn’t make an illegal move this morning” it’s “something a 1400 ranked player would never do”. My entire point is that it happens. Not often, but also not “never”.

No I definitely have, it’s just so rare I can’t remember when I last did it. I do remember playing one in a blitz tournament 20 years ago! But if this is the first game they played, or if it happens in 1/10 matches, that’s wild

Re: ChatGPT's Chess Elo is 1400

#323

Post by pattrn »

pattrn wrote: post #318
mynameisvlad wrote: post #273

Earlier quoted context omitted.

> No one is arguing that humans never make illegal moves > something a 1400 ranked player would never do > fine, fair, "never" was too much. I mean, yes they were and they said as much after I called them out on it. But go off on how nobody is arguing the literal thing that was being argued. It's not like messages are threaded or something, and read top-down. You would have 100% had to read the comment I replied to f…

You have twice removed the substance of an argument and responded to an irrelevant nitpick. Here's what the OP said: > He literally used the same prompt as the article. > Claim: "ChatGPT's Chess Elo is 1400" > Reality: ChatGPT gives illegal moves (this happened to article author too), > something a 1400 ranked player would never do > Result: ChatGPT's rank is not 1400. This is a completely fair argument that makes pe…

> Ah yes, of course, just because you never saw it means it never happens. That's definitely why rules exist around this specific thing happening. Because it never happens. Totally.

You seem to have missed the part where I said multiple times that a 1400 has definitely made illegal moves.

> In fact, it's so rare that in order to forefeit a game, you have to do it twice. But it never happens, ever, because pattrn has never seen it. Case closed everyone.

I actually said the exact opposite. You're responding to an argument I didn't make.

> I made no judgement on what ChatGPT can and can't do. I pointed out an extreme. Which the commenter agreed was an extreme. The rest of your comment is completely irrelevant but congrats on getting tilted over something that literally doesn't concern you. Next time, just save us both the time and effort and don't bother butting in with irrelevant opinions. Especially if you couldn't even bother to read what was already said.

The commenter's throwaway account never agreed it was an extreme. I agreed it was an extreme, but also that disproving that one extreme does nothing to contradict his argument. Yet again you aren't responding to the argument.

This entire exchange is baffling. You seem to be missing the point for a third time, and now you're misrepresenting what I said. Welcome to the internet, I guess.

Re: ChatGPT's Chess Elo is 1400

#324

Post by mynameisvlad »

pattrn wrote: post #323
pattrn wrote: post #318

Earlier quoted context omitted.

You have twice removed the substance of an argument and responded to an irrelevant nitpick. Here's what the OP said: > He literally used the same prompt as the article. > Claim: "ChatGPT's Chess Elo is 1400" > Reality: ChatGPT gives illegal moves (this happened to article author too), > something a 1400 ranked player would never do > Result: ChatGPT's rank is not 1400. This is a completely fair argument that makes pe…

> Ah yes, of course, just because you never saw it means it never happens. That's definitely why rules exist around this specific thing happening. Because it never happens. Totally. You seem to have missed the part where I said multiple times that a 1400 has definitely made illegal moves. > In fact, it's so rare that in order to forefeit a game, you have to do it twice. But it never happens, ever, because pattrn has…

> The commenter's throwaway account never agreed it was an extreme.

> fine, fair, "never" was too much.

This is the second time I've had to do this. Do you just pretend things weren't said or do you actually have trouble reading the comments that have been here for hours? You make these grand assertions which are disproven by... reading the things that are directly above your comment.

> This entire exchange is baffling.

Yeah your inability to read comments multiple times in a row is extremely baffling.

As I said before:

> Next time, just save us both the time and effort and don't bother butting in with irrelevant opinions. Especially if you couldn't even bother to read what was already said.

Re: ChatGPT's Chess Elo is 1400

#325

Post by serverholic »

arrrg wrote: post #274
tracker1 wrote: post #254

Earlier quoted context omitted.

How many 1400 human chess players do you have to explain every possible move to it every single move?

Does that matter? I’m really very confused by the argument you are making. That you may have to babysit this particular aspect of playing the game seems quite irrelevant to me.

[dead]

Re: ChatGPT's Chess Elo is 1400

#326

Post by throwwwaway69 »

whimsicalism wrote: post #269
pattrn wrote: post #261

Earlier quoted context omitted.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Think blindfolded 1400 players, which is what this effectively is, would make illegal moves. But even if it doesn't play like human 1400 players, if it can get to a 1400 elo while resigning games it makes illegal moves on, that seems 1400 level to me. And i bet that some 1400s do occasionally make illegal moves (missing pins) while playing otb

This isn't really an apt metaphor. Firstly because higher level blindfolded players, when trained to play with a blindfold, also virtually never make mistakes. Secondly because a computer has permanent concrete state management (compared to humans) and can, without error, keep a perfect representation of a chess if it chooses to do so.

Re: ChatGPT's Chess Elo is 1400

#327

Post by throwwwaway69 »

DSMan195276 wrote: post #286
pattrn wrote: post #261

Earlier quoted context omitted.

I think his point is that 1400 level players don't make illegal moves, therefore ChatGPT is not playing at the level of a 1400 level player.

Personally I think the illegal moves are irreverent, the fact that it doesn't play exactly like a typical 1400 doesn't mean it can't have a 1400 rating. Rating is purely determined by wins and losses against opponents, it doesn't matter if you lose a game by checkmate, resignation, or playing an illegal move. That's not to say ChatGPT can play at 1400, just that that playing in an odd way doesn't determine its rating…

This is like saying I play at a 2900 level if you just ignore all the times I lose.

Re: ChatGPT's Chess Elo is 1400

#328

Post by serverholic »

Spivak wrote: post #275
tracker1 wrote: post #254

Earlier quoted context omitted.

How many 1400 human chess players do you have to explain every possible move to it every single move?

I feel like we have very different expectations about what tools like this are good for and how to use them. When I say GPT3 can play chess what I mean is, I can build a chess playing automaton where the underlying decision making system is entirely powered by the LLm. I, as the developer, am providing contextual information like what the current board state is, and what the legal moves are, but my code doesn't actua…

[dead]

Re: ChatGPT's Chess Elo is 1400

#329

Post by bsaul »

root_axis wrote: post #248
chpatrick wrote: post #170

Earlier quoted context omitted.

"Statistical power" isn't some magic property of GPT4. It can produce statistically more likely moves because somewhere deep down it can model chess.

It isn't a model of chess, it's a model of internet text, if it was a model of chess it wouldn't make illegal moves.

if it didn't have at least some kind of model of chess, it wouldn't be able to play past midgame.

Simply because on a new position, moves from other positions aren't applicable at all.

Re: ChatGPT's Chess Elo is 1400

#330

Post by throwwwaway69 »

pattrn wrote: post #323
pattrn wrote: post #318

Earlier quoted context omitted.

You have twice removed the substance of an argument and responded to an irrelevant nitpick. Here's what the OP said: > He literally used the same prompt as the article. > Claim: "ChatGPT's Chess Elo is 1400" > Reality: ChatGPT gives illegal moves (this happened to article author too), > something a 1400 ranked player would never do > Result: ChatGPT's rank is not 1400. This is a completely fair argument that makes pe…

> Ah yes, of course, just because you never saw it means it never happens. That's definitely why rules exist around this specific thing happening. Because it never happens. Totally. You seem to have missed the part where I said multiple times that a 1400 has definitely made illegal moves. > In fact, it's so rare that in order to forefeit a game, you have to do it twice. But it never happens, ever, because pattrn has…

> The commenter's throwaway account never agreed it was an extreme.

I did, two hours ago, 6 minutes after your comment

https://news.ycombinator.com/item?id=35201830

Post reply on HN