Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 model sweat over a Paul Morphy mate-in-2

alexop.dev

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#21

Post by Gimpei »

freediver wrote: post #5

On a similar note, I just updated LLM Chess Puzzles repo [1] yesterday. The fact that gpt-4.5 gets 85% correctly solved is unexpected and somewhat scary (if model was not trained on this). [1] https://github.com/kagisearch/llm-chess-puzzles

Given that o3 is trained on the contents of the Internet, and the answers to all these chess problems are almost certainly on the Internet in multiple places, in a sense it has been weakly trained on this content. The question for me becomes: is the LLM doing better on these problems because it’s improving in reasoning, or is it simply improving in information retrieval.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#22

Post by Kapura »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable than we created in the past.

"Well, it's not a chess engine so its impressive it-" No. Stop. At best what we have here is an extremely computationally expensive way to just google a problem. We've been googling things since I was literally a child. We've had voice search with google for, idk, a decade+. A computer that can't even solve its own chess problems is an expensive regression.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#24

Post by andoando »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle?

The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook has to block, and you can see bishop either moves or captures and pawn moving forward is mate

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#26

Post by mhh__ »

Kapura wrote: post #22
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

If you mean write code to exhaustively search the solution space then they actually can do that quite happily provided you tell it you will execute the code for them

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#27

Post by bobsmooth »

Kapura wrote: post #22
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

A computer program that has the agency to google a problem, interpret the results, and respond to a human was science fiction just 10 years ago. The entire field of natural language processing has been solved and it's insane.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#28

Post by bubblyworld »

andoando wrote: post #24
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

Im 1600 rated player and this took me 20 seconds to solve, is this really considered a very hard puzzle? The obvious moves dont work, you can see whites pawn moving forward is mate, and you can see black is essentially trapped and has very limited moves, so immediately I thought first move is a waiting move and theres only two options there. Block the black pawn moving and if bishop moves, rook takes is mate. So rook…

Agreed, I'm similar fide (not rated but ~2k lichess) and it took me a few seconds as well. Not a hard puzzle, for a regular chess player anyway.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#29

Post by jncfhnb »

Kapura wrote: post #22
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

Looks to me like it would have simulated the steps using sensible tools but didn’t know it was sandboxed out of using those tools? I think that’s pretty reasonable.

Suppose we removed its ability to google and it conceded to doing the tedium of writing a chess engine to simulate the steps. Is that “better” for you?

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#30

Post by currymj »

Kapura wrote: post #22
IanCal wrote: post #18

Earlier quoted context omitted.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

> "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out"

from the article:

"3. Attempt to Use Python When pure reasoning was not enough, o3 tried programming its way out of the situation.

“I should probably check using something like a chess engine to confirm.” (tries to import chess module, but fails: “ModuleNotFoundError”).

It wanted to run a simulation, but of course, it had no real chess engine installed."

this strategy failed, but if OpenAI were to add "pip install python-chess" to the environment, it very well might have worked. in any case, the machine did exactly the thing you claim it should have done.

possibly scrolling down to read the full article makes you a rube though.

Post reply on HN