Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

331–340 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#331

Post by meh8881 »

mrguyorama wrote: post #146
LeanderK wrote: post #109

Earlier quoted context omitted.

by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.

It cannot be learning the general rules because it occasionally tries to invent pieces out of whole cloth.

Try asking it if a previous move was legal. Or why it decided to do the move.

The problem is not a failure to understand the rules. It is just not very good at maintaining the state.

Re: ChatGPT's Chess Elo is 1400

#332

Post by meh8881 »

SanderNL wrote: post #62
Jensson wrote: post #43

Earlier quoted context omitted.

We have evidence right here that patterns of moves leads to 1400. To me this is evidence that 1400 level play is very rigid and mostly relies on defined openings and strategies, making it easy for an LLM to find similar games and win using standard continuations. Just like grammar the patterns are too hard for humans to see and encode, but LLMs can encode pretty complex patterns. Domains that are easy to encode as gr…

At what point can we just say that understanding “patterns of moves” is understanding chess? It seems you suggest there is more to it, but maybe I am mistaken.

Ask it to evaluate moves and explain why they are or are not good

Re: ChatGPT's Chess Elo is 1400

#333

Post by whimsicalism »

throwwwaway69 wrote: post #326
whimsicalism wrote: post #269

Earlier quoted context omitted.

Think blindfolded 1400 players, which is what this effectively is, would make illegal moves. But even if it doesn't play like human 1400 players, if it can get to a 1400 elo while resigning games it makes illegal moves on, that seems 1400 level to me. And i bet that some 1400s do occasionally make illegal moves (missing pins) while playing otb

This isn't really an apt metaphor. Firstly because higher level blindfolded players, when trained to play with a blindfold, also virtually never make mistakes. Secondly because a computer has permanent concrete state management (compared to humans) and can, without error, keep a perfect representation of a chess if it chooses to do so.

1400 FIDE !=. high level blindfolded player.

Re: ChatGPT's Chess Elo is 1400

#334

Post by meh8881 »

Reptur wrote: post #37

It can't even play tic tac toe without making mistakes. My prompt told it to play a perfect game, predicting all next possible moves, the goal is stalemate or win. It would still make sub-optimal moves and fail at this. Even when I explained what went wrong, it would then apologize and do another wrong move.

Why would you think telling it to play a perfect game, predicting all next possible moves, would help?

That definitely made it worse.

Re: ChatGPT's Chess Elo is 1400

#335

Post by sebzim4500 »

z3c0 wrote: post #227
vidarh wrote: post #212

Earlier quoted context omitted.

You're "disproving" the article by doing things differently to how the article did. If you're going to disprove that the method given in the article does as well as the article claims at least use the same method.

They are disproving an assertion. Demonstrating that an alternate approach implodes the assertion is a perfectly acceptable route, especially when the original approach was cherry-picking successes and throwing out failures. I wish I could just make bullshit moves and get a higher chess ranking. Sounds nice.

He claims he was forfeiting every time he got an illegal move. Does no one on this website actually read the article?

Whether any of it is actually true is a different question.

Re: ChatGPT's Chess Elo is 1400

#336

Post by pattrn »

throwwwaway69 wrote: post #330
pattrn wrote: post #323

Earlier quoted context omitted.

> Ah yes, of course, just because you never saw it means it never happens. That's definitely why rules exist around this specific thing happening. Because it never happens. Totally. You seem to have missed the part where I said multiple times that a 1400 has definitely made illegal moves. > In fact, it's so rare that in order to forefeit a game, you have to do it twice. But it never happens, ever, because pattrn has…

> The commenter's throwaway account never agreed it was an extreme. I did, two hours ago, 6 minutes after your comment https://news.ycombinator.com/item?id=35201830

Thanks! I appreciate it.

Re: ChatGPT's Chess Elo is 1400

#337

Post by sebzim4500 »

tracker1 wrote: post #254
Spivak wrote: post #213

Earlier quoted context omitted.

From the article. > Occasionally it does make an illegal move, but I decided to interpret that as ChatGPT flipping the table and saying “this game is impossible, I literally cannot conceive of how to win without breaking the rules of chess.” So whenever it wanted to make an illegal move, it resigned. But you can do even better than the OP with a few tweaks. 1. One is by taking the most common legal move from a sample…

How many 1400 human chess players do you have to explain every possible move to it every single move?

When they are blindfolded? Almost all of them.

Re: ChatGPT's Chess Elo is 1400

#338

Post by singularity2001 »

teej wrote: post #303

I just deployed a GPT-4 powered chess bot to lichess. You can challenge it here: https://lichess.org/@/oopsallbots-gpt-4

What happens if you add the rules to the initial prompt and something like: "Think step by step, explain your move and if it is illegal rethink"

Re: ChatGPT's Chess Elo is 1400

#339

Post by kdmccormick »

qwytw wrote: post #98
psychphysic wrote: post #85

Earlier quoted context omitted.

That's how one uses any tool.

The behavior of pretty much every other tool is much easier to interpret though.

If the title of the article was:

> A trivial wrapper around ChatGPT has a Chess Elo of 1400

would you have any issue?

Afaict, the thesis is the article is not "ChatGPT is the ideal tool for playing AI chess", but "it is interesting how well ChatGPT can play chess with some very simple tweaks."

Re: ChatGPT's Chess Elo is 1400

#340

Post by lostmsu »

liampulles wrote: post #263

With respect to the points here on how illegal moves invalidate ChatGPT as a valid Chess AI, consider the following: AI.v1: Feed ChatCPT the move history and ask for the next move. AI.v2: Wrap AI.v1 with additional logic to try and translate any invalid response as a similar/close valid move, else treat it as a resignation. I would say AI.v2 is a "valid" Chess AI since it obeys the rules of chess. Also I think it is…

With chatGPT you can also tell it the move is illegal and let it try again.
Post reply on HN