Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 model sweat over a Paul Morphy mate-in-2

alexop.dev

Post reply on HN
Print view View original

51–60 of 73 posts

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#51

Post by toolslive »

simonw wrote: post #48
toolslive wrote: post #46

Earlier quoted context omitted.

or just present it with the list of legal moves and force it to pick from said list.

I imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.

Nope. The list is very limited. For the starting position: a3, a4, b3,b4,.......h3, h4, Na3, Nc3, Nf3, Nh3

That's 20 moves. the size grows a bit in the early middle game, but then drops again in the endgame. There do exist rather artificial positions with more than 200 legal moves, but the average number of legal moves in a position is around 40.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#52

Post by dmurray »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

> This is also a chess puzzle marked as very hard that a person who is good at chess should give themselves fifteen minutes to solve. The author of the chess.com blog containing this puzzle only solved about half of them!

I am human and I solved this before opening the blog post, because I've seen this problem 100 times before with this exact description. I don't understand why an LLM wouldn't have done the same, because pattern matching off things you saw on the internet is IIUC the main way LLMs work.

(I am good at chess, but not world class. This is not a difficult mate in 2 problem: if I hadn't seen it, it would take a minute or so to solve, some composed 2-movers might take me 5 minutes).

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#53

Post by dmurray »

dmurray wrote: post #52
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

> This is also a chess puzzle marked as very hard that a person who is good at chess should give themselves fifteen minutes to solve. The author of the chess.com blog containing this puzzle only solved about half of them! I am human and I solved this before opening the blog post, because I've seen this problem 100 times before with this exact description. I don't understand why an LLM wouldn't have done the same, bec…

I just tried ChatGPT free with the prompt "There's a mate-in-two composed by Paul Morphy. What's the key move?". It searches and finds it immediately. But if I ask it not to search the internet, its response is incoherent (syntactically valid English and knows the names of the chess pieces, but otherwise hallucinated).

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#54

Post by Hammershaft »

freediver wrote: post #5

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday. The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if model was not trained on this). [1] https://github.com/kagisearch/llm-chess-puzzles

It's getting incredibly difficult to find anything on the internet that these models weren't trained on, which is why recent llm tests have used so much secrecy and only shows a few sample questions.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#56

Post by red369 »

Shorn wrote: post #43

I asked ChatGPT about playing chess: it says tests have shown it makes an illegal move within 10 - 15 moves, even if prompted to play carefully and not make any illegal moves. It'll fail within the first 3 or 4 if you ask it play reasonably quickly. That means, it can literally never win a chess match, given an intentional illegal move is an immediate loss. It can't beat a human who can't play chess. It literally can…

I have tried playing chess with ChatGPT a couple of times recently, and I found it was making illegal moves after about 4 or 5 moves.

The first few could be resolved by asking it to check its moves. After a few more, I was having to explain that knights can jump and therefore can’t be blocked. It was also trying to move pieces that weren’t there, onto squares alert occupied by its own pieces, and asking it to review was not getting anywhere. 10-15 moves is very optimistic, unless it’s counting each move by either side, i.e., White moves 5-8 times and Black moves 5-8 times. Even that seems optimistic, but the lower end could be right.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#57

Post by IanCal »

andoando wrote: post #24
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle? The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook…

I don't know, I didn't spot the answer and it's from a list of hard puzzles from a chess coach. The model also wasn't told it was mate in 2 (or even if a mate was possible), just to solve it and it was white to move.

https://www.chess.com/blog/ThePawnSlayer/checkmate-in-two-pu...

Although perhaps this is missing the point - the process and chain here in response to an image and a sentence is extremely impressive. You can argue it's not useful, or not useful for specific use cases but it's impressive.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#58

Post by IanCal »

foundry27 wrote: post #33

I just tried the same puzzle in o3 using the same image input, but tweaked the prompt to say “don’t use the search tool”. Very similar results! It spent the first few minutes analyzing the image and cross-checking various slices of the image to make sure it understood the problem. Then it spent the next 6-7 minutes trying to work through various angles to the problem analytically. It decided this was likely a mate-in…

Told that it was a mate in 2 puzzle, and it solved it for me

https://chatgpt.com/share/680f4a02-4cc4-8002-8301-59214fca78...

It worked through some stuff then decided to try and list all possible moves as there can't be that many. Tried importing stuff that didn't work, then wrote code to create the permutations.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#59

Post by stavros »

toolslive wrote: post #51
simonw wrote: post #48

Earlier quoted context omitted.

I imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.

Nope. The list is very limited. For the starting position: a3, a4, b3,b4,.......h3, h4, Na3, Nc3, Nf3, Nh3 That's 20 moves. the size grows a bit in the early middle game, but then drops again in the endgame. There do exist rather artificial positions with more than 200 legal moves, but the average number of legal moves in a position is around 40.

You could also constrain the output grammar to legal moves, but if we're comparing its chess performance to humans', it would be unfair to not let it think.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#60

Post by red369 »

red369 wrote: post #56
Shorn wrote: post #43

I asked ChatGPT about playing chess: it says tests have shown it makes an illegal move within 10 - 15 moves, even if prompted to play carefully and not make any illegal moves. It'll fail within the first 3 or 4 if you ask it play reasonably quickly. That means, it can literally never win a chess match, given an intentional illegal move is an immediate loss. It can't beat a human who can't play chess. It literally can…

I have tried playing chess with ChatGPT a couple of times recently, and I found it was making illegal moves after about 4 or 5 moves. The first few could be resolved by asking it to check its moves. After a few more, I was having to explain that knights can jump and therefore can’t be blocked. It was also trying to move pieces that weren’t there, onto squares alert occupied by its own pieces, and asking it to review…

I just tried again, and ChatGPT did much better. A notification said it was using GPT-4o mini, and it reached move 10 for White (me) before it lost the plot:

https://chatgpt.com/share/680f57b6-8554-800b-a042-f640224b91...

It didn't get much further with suggestions to review. Also, the small ASCII board it generated was incorrect much earlier, but it sometimes plays without that, so I let that go.

Post reply on HN