Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Watching o3 model sweat over a Paul Morphy mate-in-2

alexop.dev

Post reply on HN
Print view View original

41–50 of 73 posts

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#41

Post by alexop »

CSMastermind wrote: post #37

It's weird to me that the author says this behavior feels human because it's nothing like how I solve this puzzle. At no point during my process would I be counting pixels in the image. It feels very clearly like a machine that mimics human behavior without understanding where that behavior comes from.

Yes, exactly. What I meant is that a human would also try every "tool" available. In the case of o3, the only tools it had were Python and Bing.

But you are right. It does not actually understand anything. It is just a next-token predictor that happens to have access to Python and Bing.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#42

Post by otabdeveloper4 »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

OpenAI is a commercial company and their product is to make anthropomorphic chat bots.

Clever Hans at web-scale, so to say.

So if you're impressed by a model that spent 10 minutes and single-digit dollars to not solve a problem that has been solved before, then I guess their model is working exactly as expected.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#43

Post by Shorn »

I asked ChatGPT about playing chess: it says tests have shown it makes an illegal move within 10 - 15 moves, even if prompted to play carefully and not make any illegal moves. It'll fail within the first 3 or 4 if you ask it play reasonably quickly.

That means, it can literally never win a chess match, given an intentional illegal move is an immediate loss.

It can't beat a human who can't play chess. It literally can't even lose properly. It will disqualify itself every time.

--

> It shows clearly where current models shine (problem-solving)

Yeh - that's not what's happening.

I say that as someone that pays for and uses an LLM pretty much every day.

--

Also - I didn't fact check any of the above about playing chess. I choose to believe.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#44

Post by otabdeveloper4 »

bobsmooth wrote: post #27
Kapura wrote: post #22

Earlier quoted context omitted.

I am sorry, but if this impresses you you are a rube. If this were a machine with the smallest bit of actual intelligence it would, upon seeing its a chess puzzle, remember "hey, i am a COMPUTER and a small set of fixed moves should take me about 300ms or so to fully solve out" and then do that. If the machine _literally has to cheat to solve the puzzle_ then we have made technology that is, in fact, less capable tha…

A computer program that has the agency to google a problem, interpret the results, and respond to a human was science fiction just 10 years ago. The entire field of natural language processing has been solved and it's insane.

OpenAI's whole business is impressing you with whiz-bang sci-fi sound and fury.

This is a bad thing because it means they gave up on solving actual problems and entered the snake oil business.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#45

Post by simonw »

Shorn wrote: post #43

I asked ChatGPT about playing chess: it says tests have shown it makes an illegal move within 10 - 15 moves, even if prompted to play carefully and not make any illegal moves. It'll fail within the first 3 or 4 if you ask it play reasonably quickly. That means, it can literally never win a chess match, given an intentional illegal move is an immediate loss. It can't beat a human who can't play chess. It literally can…

Preventing an LLM from making illegal moves should be very simple: provide it with tool access to something that tells it if a move is legal or not, then watch it iterate in a loop until it finds a move that it is allowed to make.

I expect this would dramatically improve the chess playing abilities of the competent tool using models, such as O3.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#46

Post by toolslive »

simonw wrote: post #45
Shorn wrote: post #43

I asked ChatGPT about playing chess: it says tests have shown it makes an illegal move within 10 - 15 moves, even if prompted to play carefully and not make any illegal moves. It'll fail within the first 3 or 4 if you ask it play reasonably quickly. That means, it can literally never win a chess match, given an intentional illegal move is an immediate loss. It can't beat a human who can't play chess. It literally can…

Preventing an LLM from making illegal moves should be very simple: provide it with tool access to something that tells it if a move is legal or not, then watch it iterate in a loop until it finds a move that it is allowed to make. I expect this would dramatically improve the chess playing abilities of the competent tool using models, such as O3.

or just present it with the list of legal moves and force it to pick from said list.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#48

Post by simonw »

toolslive wrote: post #46
simonw wrote: post #45

Earlier quoted context omitted.

Preventing an LLM from making illegal moves should be very simple: provide it with tool access to something that tells it if a move is legal or not, then watch it iterate in a loop until it finds a move that it is allowed to make. I expect this would dramatically improve the chess playing abilities of the competent tool using models, such as O3.

or just present it with the list of legal moves and force it to pick from said list.

I imagine there are points in a chess game, especially early on, where that list could have hundreds of moves - could use up a fair amount of tokens.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#49

Post by nathell »

IanCal wrote: post #18
Kapura wrote: post #17

So... it failed to solve the puzzle? That seems distinctly unimpressive, especially for a puzzle with a fixed start state and a limited set of possible moves.

> That seems distinctly unimpressive I cannot understate how impressive this is to me, having been involved in ai research projects and robotics in years gone by. This is a general purpose model, given an image and human written request that then step by step analyses the image, iterates through various options, tries to write code to solve the problem and then searches the internet for help. It reads multiple result…

> This is also a chess puzzle marked as very hard that a person who is good at chess should give themselves fifteen minutes to solve.

Is it, though? I play at around 1000 Elo – I have a long-standing interest in chess, but my brain invariably turns on fog of war that makes me not notice threats to my queen or something – and I solved it in something like one minute. It has very little moving parts, so the solution, while beautifully unobvious, can be easily brute-forced by a human.

Re: Watching o3 model sweat over a Paul Morphy mate-in-2

#50

Post by cess11 »

"o3 does not just spit out an answer. It reasons. It struggles. It switches tools. It self-corrects. Sometimes it even cheats, but only after exhausting every other option. That feels very human."

I've never met a human player that suddenly says 'OK, I need Python to figure out my next move'.

I'm not a good player, usually I just do ten minute matches against the weakest Stockfish settings so as not to be annoying to a human, and I figured this one out in a couple of minutes because there are very few options. Taking with the rook doesn't work, taking with the pawn also doesn't, so it has to be a non-taking move, and the king can't do anything useful so it has to be the rook and typically in these puzzles it's a sacrifice that unlocks the solution. And it was.

Post reply on HN