Earlier quoted context omitted.
This is not a good analogy because reasoning models are not choosing the best from a set of attempts based on knowledge of the correct answer. It really is more like what it sounds like: “did you think about it longer until you ruled out various doubts and became more confident?” Of course nobody knows quite why directing more computation in this way makes them better, and nobody seems to take the reasoning trace too…
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#101They experimented with gpt-2 scale models. Hard to make any meaningful conclusions in the gpt-5 era.