Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 model sweat over a Paul Morphy mate-in-2

alexop.dev

Post reply on HN
Print view View original

31–40 of 73 posts

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#31

Post by kamranjon »

alexop wrote: post #19
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Yes, I agree. Like I said, in the end it did what a human would do: google for the answer. Still, it was interesting to see how the reasoning unfolded. Normally, humans train on these kinds of puzzles until they become pure pattern recognition. That's why you can't become a grandmaster if you only start learning chess as an adult — you need to be a kid and see thousands of these problems early on, until recognizing t…

I am a human and I figured this puzzle out in under a minute by just trying the small set of possible moves until I got it correct. I am not a serious chess player. I would have expected it to at least try the possible moves? I think this maybe lends credence to the idea that these models aren’t actually reasoning but are doing a great job of mimicking what we think humans do.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#32

Post by [deleted] »

andoando wrote: post #24
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle? The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook…

[deleted]

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#33

Post by foundry27 »

I just tried the same puzzle in o3 using the same image input, but tweaked the prompt to say “don’t use the search tool”. Very similar results!

It spent the first few minutes analyzing the image and cross-checking various slices of the image to make sure it understood the problem. Then it spent the next 6-7 minutes trying to work through various angles to the problem analytically. It decided this was likely a mate-in-two (part of the training data?), but went down the path that the key to solving the problem would be to convert the position to something more easily solvable first. At that point it started trying to pip install all sorts of chess-related packages, and when it couldn’t get that to work it started writing a simple chess solver in Python by hand (which didn’t work either). At one point it thought the script had found a mate-in-six that turned out to be due to a script bug, but I found it impressive that it didn’t just trust the script’s output - instead it analyzed the proposed solution and determined the nature of the bug in the script that caused it. Then it gave up and tried analyzing a bit more for five more minutes, at which point the thinking got cut off and displayed an internal error.

15 minutes total, didn’t solve the problem, but fascinating! There were several points where if the model were more “intelligent”, I absolutely could see it reasoning it out following the same steps.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#34

Post by dimatura »

bobsmooth wrote: post #27
Kapura wrote: post #22

Earlier quoted context omitted.

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

A computer program that has the agency to google a problem, interpret the results, and respond to a human was science fiction just 10 years ago. The entire field of natural language processing has been solved and it's insane.

Honestly, I think that if in 2020 you had asked me whether we would be able to do this in 2025, I would've guessed no, with a fairly high confidence. And I was aware of GPT back then.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#37

Post by CSMastermind »

It's weird to me that the author says this behavior feels human because it's nothing like how I solve this puzzle.

At no point during my process would I be counting pixels in the image. It feels very clearly like a machine that mimics human behavior without understanding where that behavior comes from.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#38

Post by aledalgrande »

On a sidenote, I tried to get Codex + O3 to make an existing sidebar toggable with Tailwind CSS and it made an abomination full of bugs. This is a classic "boilerplate" task I'd expect it to be able to do. Not sure if I'm doing it wrong but... a little bit more direct instructions to O4-mini and it managed. The cost was astronomical tho compared to Anthropic.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#39

Post by triyambakam »

bfung wrote: post #25

LLMs are not chess engines, similar to how they don’t really calculate arithmetic. What’s new? carry on.

Yeah it's rather annoying how people (maybe due to marketing) expect a generalized model to be able to be an expert in every domain.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#40

Post by globnomulous »

Gimpei wrote: post #21
freediver wrote: post #5

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday. The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if model was not trained on this). [1] https://github.com/kagisearch/llm-chess-puzzles

Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.

And then there's the further question of where we draw the line in ourselves. One of my teachers -- a philosopher -- once said that real, actual thought is incredibly rare. He's a world-renowned expert but says he can count on one hand the number of times in his life that he felt he was thinking rather than remembering and reorganizing what he already knew.

That's not to say "are you remembering or reasoning" means the same thing when applied to humans vs when it's applied to LLMs.

Post reply on HN