Better than me then.
But does it give credit to who taught it. These models are basically a scrape of the best of humankind and a claim that it's their own.
ChatGPT's Chess Elo is 1400
31–40 of 361 posts
Re: ChatGPT's Chess Elo is 1400
#32When I tried it at v3.0 i found after 5-10 moves it started moving illegally.
Did you use the same prompt method as OP?
Re: ChatGPT's Chess Elo is 1400
#33Earlier quoted context omitted.
You don’t get to 1400 like that. The amount of moves it has to literally remember is stupendous.
It doesn't remember exact moves, it remembers patterns of moves and continues those. That is why it so easily makes illegal moves, because the patterns doesn't have anything to do with board state.
Again, not enough for 1400. 800 maybe.
Re: ChatGPT's Chess Elo is 1400
#34Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.
This is actually a huge debate right now.
OpenAI is on the side of 'LLMs have only surprised us to the upside, so using crutches is counterproductive'
Whereas other people think 'Teaching an LLM to do arbitrary math problems through brute force is probably one of the most wasteful things imaginable when calculators exist.'
I'm actually very excited to see which side wins (I'm on team calculator, but want to be on team LLM)
Re: ChatGPT's Chess Elo is 1400
#35Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
> Most likely it has seen a similar sequence of moves in its training set.
Wouldn't we expect a much higher rate of illegal moves if that was the case?
Re: ChatGPT's Chess Elo is 1400
#36Chessnetwork did a video about this a few weeks ago FYI - https://www.youtube.com/watch?v=svlIYFpsWs0
Seems its strength is mostly in memorisation of the book openings. Still impressive nonetheless.
Re: ChatGPT's Chess Elo is 1400
#37It can't even play tic tac toe without making mistakes. My prompt told it to play a perfect game, predicting all next possible moves, the goal is stalemate or win. It would still make sub-optimal moves and fail at this. Even when I explained what went wrong, it would then apologize and do another wrong move.
Re: ChatGPT's Chess Elo is 1400
#38Earlier quoted context omitted.
Nobody who is 1400 plays outright illegal moves.
Humans don’t, but that is an unnecessary bar IMO.
> Humans don’t, but that is an unnecessary bar IMO.
People are always telling me that I'm moving the goalposts when I challenge the hyperbole about LLMs. But now you're moving the goalposts about chess.
Not playing illegal moves is a pre-requisite for any strong understanding of how to play chess. That is definitely the goal post.
Re: ChatGPT's Chess Elo is 1400
#39Why not just introduce AlphaGo as an API that can be used by chatGPT? So every time you want to do a this type of gaming, you just send a request. I mean, chatGPT sends a request to AlphaGo, but as a user you don't know actually what's happening. But in the background, it happens really fast, so it's just like you are chatting with chatGPT, but using much, much powerful tool to do this kind of things.
That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.
What if you come up with a prompt to tell ChatGPT to figure out how to use APIs and external tools to help itself get better at chess?
Re: ChatGPT's Chess Elo is 1400
#40Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.
> Most likely it has seen a similar sequence of moves in its training set. Wouldn't we expect a much higher rate of illegal moves if that was the case?
If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.