Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT's Chess Elo is 1400

dkb.blog

Post reply on HN
Print view View original

101–110 of 361 posts

Re: ChatGPT's Chess Elo is 1400

#101

Post by paxys »

armchairhacker wrote: post #73
paxys wrote: post #24

Earlier quoted context omitted.

That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.

How about, can a generative AI teach itself how to use a chess AI to beat chess? Give GPT4 the ability to make REST API calls and also access to FFI, and put a chess-bot library somewhere. Train it how to use these but not necessarily how to use the chess API specifically. If you ask GPT4 to play chess, can it call into that library and use the requests/responses? This has bigger ramifications too: if GPT4 learns how…

GPT is a language model. It doesn't call APIs. You can take the output of GPT and decide to call a specific API based on it, and sure you can bake that into ChatGPT or some independent tooling, but again that's not the purpose of the core project. I'm sure plenty of people/companies are working on "chess engine interface using ChatGPT" already.

Re: ChatGPT's Chess Elo is 1400

#102

Post by layer8 »

stagger87 wrote: post #58
paxys wrote: post #24

Earlier quoted context omitted.

That kinda defeats the purpose. Of course you can use AlphaGo, but the question here is – can a generative AI teach itself to play chess (and do a million other similar generic tasks) when given no specific training for it.

Honest questions, Isn't it very likely chess games were part of the training data? If so, chess sequences seem like an amazing matched filter of sorts. It doesn't seem surprising that it can guess a next move. Is it teaching itself anything here? Is the model being updated in any way? Or are you talking more generally, looking into the future sort of statement?

I think they meant “teach itself” in the course of a conversation, within the token limit. ChatGPT will forget what it has “learned” once the conversation is over.

Re: ChatGPT's Chess Elo is 1400

#103

Post by GauntletWizard »

johnfn wrote: post #50
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

There's a pretty finite set of tokens that represent valid chess moves, and a more than countable set of possible opening moves, with a very distinct probability of moves; Nobody opens pg3, for example.

Classical Markov chains played chess at some rate of success. ChatGPT is probably a lot better but not fundamentally different - It's predicting which moves to play based on sets of past games, not by memorizing it but by memoizing it.

Re: ChatGPT's Chess Elo is 1400

#104

Post by dragontamer »

oezi wrote: post #86
WoodenChair wrote: post #40

Earlier quoted context omitted.

If there was an actual understanding of chess at a 1400 level we wouldn't expect any illegal moves.

I think there is very low percentage of players at elo 1400 who can provide a valid next move after seeing just the list of moves and not the current board state.

I'm Elo 1400 and can beat literally everyone I know in the real world. I need to go online to find players at my skill level, or find tournament/competitive settings for a challenge.

Yeah, I'm "class C", weak amateur chess player, but I think you're grossly underestimating the amount of study I put into this game. I'm not going to make an illegal move

Re: ChatGPT's Chess Elo is 1400

#105

Post by scj »

WoodenChair wrote: post #38
SanderNL wrote: post #29

Earlier quoted context omitted.

Humans don’t, but that is an unnecessary bar IMO.

> Humans don’t, but that is an unnecessary bar IMO. People are always telling me that I'm moving the goalposts when I challenge the hyperbole about LLMs. But now you're moving the goalposts about chess. Not playing illegal moves is a pre-requisite for any strong understanding of how to play chess. That is definitely the goal post.

If a two year old had an ELO of 1400 but sometimes took a nap halfway through a game, it'd be a prodigy.

It's not like an AI making silly mistakes when driving a car.

Re: ChatGPT's Chess Elo is 1400

#106

Post by famouswaffles »

Jensson wrote: post #84
famouswaffles wrote: post #64

Earlier quoted context omitted.

Not if training is unsupervised. If you've never been explicitly told the rules of game, you can never be 100% sure of all possible illegal moves. anyway the 3.5 series can't ply chess but gpt-4 certainly can.

> anyway the 3.5 series can't ply chess but gpt-4 certainly can. This article stated the opposite, gpt-4 couldn't play chess while gpt-3.5 could. So this is a case where the model got dumber.

Not sure what's up but from here https://www.reddit.com/r/chess/comments/11rfbch/gpt4_seems_t... nd here https://twitter.com/zswitten/status/1631107663500304384 Gpt-4 plys chess just fine.

Re: ChatGPT's Chess Elo is 1400

#107

Post by renewiltord »

Out_of_Characte wrote: post #100
psychphysic wrote: post #85

Earlier quoted context omitted.

That's how one uses any tool.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.

The article:

> So whenever it wanted to make an illegal move, it resigned.

You:

> By comparison, any player making an illegal move is forfeiting the game...

By comparison indeed.

Re: ChatGPT's Chess Elo is 1400

#108

Post by sebzim4500 »

Out_of_Characte wrote: post #100
psychphysic wrote: post #85

Earlier quoted context omitted.

That's how one uses any tool.

Yes, but it also completely invalidates the measurement of a 1400 elo rating. By comparison, any player making an illegal move is forfeiting the game, almost all people from ~300 elo can play without making illegal moves, chatgpt cant.

ChatGPT did forfeit whenever it made an illegal move, read the article.

Re: ChatGPT's Chess Elo is 1400

#109

Post by LeanderK »

melenaboija wrote: post #87
johnfn wrote: post #50

Earlier quoted context omitted.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

If it isn not memorizing, how do you think is doing it?

by trying to learning the general rules that to explain the dataset and minimise its loss. That's what machine learning is about, it's not called machine memorising.

Re: ChatGPT's Chess Elo is 1400

#110

Post by mtlmtlmtlmtl »

johnfn wrote: post #50
WoodenChair wrote: post #8

Most likely it has seen a similar sequence of moves in its training set. There are numerous chess sites with databases displayed in the form of web pages with millions of games in them. If it had any understanding of chess, it would never play an illegal move. It's not surprising that given a sequence of algebraic notation it can regurgitate the next move in a similar sequence of algebraic notation.

> Most likely it has seen a similar sequence of moves in its training set. Is this a joke making fun of the common way people dismiss other ChatGPT successes? This makes no sense with respect to chess, because every game is unique, and playing a move from a different game in a new game is nonsensical.

1850 ELO player and also chess AI programmer here. This is an oversimplification at best.

Many many games follow the same moves(1 move = 2 plies) for a long time, up to 30 moves in some cases, 20 moves is downright common and 10 moves is more common than not.

These series of moves are referred to as opening theory and are described at copious length in tons of books.

This is because while the raw number of possible paths to take is immense, the number of reasonable paths for 2 players of a given strength gets smaller and smaller.

If I went over the 300 or so classical tournament games I've played I would ballmark that maybe just one or two would deviate from all known theory in the first 10 moves.

So the criticism is valid in my view. The existence of copious chess literature can't simply be ignored here.

Post reply on HN