Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 model sweat over a Paul Morphy mate-in-2

alexop.dev

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#61

Post by bko »

foundry27 wrote: post #33

I just tried the same puzzle in o3 using the same image input, but tweaked the prompt to say “don’t use the search tool”. Very similar results! It spent the first few minutes analyzing the image and cross-checking various slices of the image to make sure it understood the problem. Then it spent the next 6-7 minutes trying to work through various angles to the problem analytically. It decided this was likely a mate-in…

Claude gets the right answer but misplaces the pieces in its initial analysis which means the answer is incorrect.

Whats going on? Did it just get lucky? Did it memorize the answer but misplace the pieces in its recall? Did it actually compute anything?

https://claude.ai/share/d640bc4c-8dd8-4eaa-b10b-cb3f83a6b94b

This is the board as it sees it (incorrect):

https://lichess.org/editor/kb6/pp6/2P5/8/8/3K4/8/R7_w_-_-_0_...

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#62

Post by simonw »

toolslive wrote: post #51
simonw wrote: post #48

Earlier quoted context omitted.

I imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.

Nope. The list is very limited. For the starting position: a3, a4, b3,b4,.......h3, h4, Na3, Nc3, Nf3, Nh3 That's 20 moves. the size grows a bit in the early middle game, but then drops again in the endgame. There do exist rather artificial positions with more than 200 legal moves, but the average number of legal moves in a position is around 40.

Huh, that's really interesting, thanks.

I mentally counted the starting moves as being 8 pawns x2 = 16 pawn moves and 2x2 =4 4 knight moves, but then I doubled it for both sides to get 40 (which with hindsight was obviously wrong) and then assumed that once the pawns had moved a bit there would be more options from non-pawn pieces.

With an upper bound of ~200 in edge cases listing all possible moves wouldn't take up much room in the context at all. I wonder if it would give better results, too.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#63

Post by SirMaster »

andoando wrote: post #24
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle? The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook…

Yeah I came here to say this... I don't even play chess (though I know the rules) and I solved this in a few minutes of looking at it. There is no way this is "hard" unless I simply got lucky? Not sure what the odds are of getting lucky solving a puzzle like this as I have never done a chess puzzle before.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#64

Post by scotty79 »

Interesting. Personally my thought process was like that:

- Check obvious, wrong moves.

- Ask what I need to have to win the game even if there's just black king left. Answer is I need all 3 pieces to win some day even if there's just black king on the board.

- So any moves that makes me lose my pawn or rook result in failure.

- So the only thing I can do with the rook is move it vertically. Any horizontal move allows black to take my pawn. King and pawn don't have much options and all result in pawn loss or basically skipping a turn while changing situation a little bit for the worse that makes mate in one move unlikely.

- Taking a pawn with rook results in loss of the rook which is just as bad.

- Let's look at spot next to the pawn. I'll still protect my pawn, but my rook is in danger. But if black takes rook, I can just move my pawn forward to get a mate. If they don't I can move rook forward and get a mate. Solved.

So I skipped trying to run a program and googling part, not because it didn't came to my mind but because I wanted different kind of challenge then challenge of extracting information from the internet or challenge of running a unfamiliar piece of software.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#65

Post by scotty79 »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

> This is also a chess puzzle marked as very hard that a person who is good at chess should give themselves fifteen minutes to solve.

I haven't played chess in decades and was never any good at it. I'm basically now at the level that I know most of the basic rules of the game. And it took me maybe 5 minutes.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#67

Post by legerdemain »

baby wrote: post #66

BTW can someone tell me how do you who you are here? I'm reading: > Chess Puzzle Checkmate in 2 White does it mean we are white, or does it mean we're trying to checkmate white?

You are playing white. It's your move. Describe a strategy where on your second move you declare checkmate. Your strategy can have decision branches, but no branch is longer than two moves.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#68

Post by pjob »

IanCal wrote: post #57
andoando wrote: post #24

Earlier quoted context omitted.

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle? The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook…

I don't know, I didn't spot the answer and it's from a list of hard puzzles from a chess coach. The model also wasn't told it was mate in 2 (or even if a mate was possible), just to solve it and it was white to move. https://www.chess.com/blog/ThePawnSlayer/checkmate-in-two-pu... Although perhaps this is missing the point - the process and chain here in response to an image and a sentence is extremely impressive. You…

I came away much less impressed than you did. The "step by step analysis" consists mostly of it considering, ruling out, and reconsidering an obviously invalid move. The code that it "tries to write" first zooms and pans around the image for no reason as it's already identified the layout of the pieces in the initial analysis. It then tries to import a library it has not yet installed in the sandbox (in addition to importing `chess.polyglot` for no discernable reason) before giving up on that thread entirely. It then manages to write a one-liner that contains an IndentationError before spending more time/tokens reestablishing the board layout. It does all of this before finally delegating the question to a search engine.

If you just paste the image into a search engine (without needing to include the text prompt) the first result contains the solution. We live in a world where Sam Altman claims that usage of words like "please" and "thank you" in prompts have cost OpenAI "tens of millions of dollars"[0]. In this case, OpenAI's "most powerful reasoning model"[1] spends 7m 51s churning through expensive output tokens spinning its wheels before ultimately giving up and searching the internet. This strikes me as incredibly wasteful. It feels like the LLM equivalent of "punch[ing] through the table". The most impressive thing to me here is that OpenAI is getting people to pay for all this nonsense.

[0] https://www.usatoday.com/story/tech/2025/04/22/please-thank-...

[1] https://platform.openai.com/docs/models/compare

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#69

Post by bluefirebrand »

globnomulous wrote: post #40
Gimpei wrote: post #21

Earlier quoted context omitted.

Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.

And then there's the further question of where we draw the line in ourselves. One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare. He's a world-renowned expert but says he can count on one hand the number of times in his life that he felt he was thinking rather than remembering and reorganizing what he already knew. That's not to say "are you remembering or reasoning" means t…

> One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare

Probably should listen to psychologists and neuroscientists about this, not philosophers tbh

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#70

Post by bluefirebrand »

simonw wrote: post #48
toolslive wrote: post #46

Earlier quoted context omitted.

or just present it with the list of legal moves and force it to pick from said list.

I imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.

At any given time there are way more illegal moves than legal moves, though
Post reply on HN