I've definitely seen some unexpected behavior from gpt5. For example, it will tell me my query is banned and then give me a full answer anyway.
From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
41–50 of 102 posts
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#42I have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?
OK, we've removed all understanding from the title above.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#43Earlier quoted context omitted.
This is not a good analogy because reasoning models are not choosing the best from a set of attempts based on knowledge of the correct answer. It really is more like what it sounds like: “did you think about it longer until you ruled out various doubts and became more confident?” Of course nobody knows quite why directing more computation in this way makes them better, and nobody seems to take the reasoning trace too…
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
I keep wondering whether people have actually examined how this work draws its conclusions before citing it.
This is science at its worst, where you start at an inflammatory conclusion and work backwards. There is nothing particularly novel presented here, especially not in the mathematics; obviously performance will degrade on out-of-distribution tasks (and will do so for humans under the same formulation), but the real question is how out-of-distribution a lot of tasks actually are if they can still be solved with CoT. Yes, if you restrict the dataset, then it will perform poorly. But humans already have a pretty large visual dataset to pull from, so what are we comparing to here? How do tiny language models trained on small amounts of data demonstrate fundamental limitations?
I'm eager to see more works showing the limitations of LLM reasoning, both at small and large scale, but this ain't it. Others have already supplied similar critiques, so let's please stop sharing this one around without the grain of salt.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#44I have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?
Do you yourself really understand, or are you just depolarizing neurons that have reached their threshold?
It can be simultaneously true that human understanding is just a firing of neurons but that the architecture and function of those neural structures is vastly different than what an LLM is doing internally such that they are not really the same. Encourage you to read Apple’s recent paper on thinking models; I think it’s pretty clear that the way LLMs encode the world is drastically inferior to what the human brain does. I also believe that could be fixed with the right technical improvements, but it just isn’t the case today.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#45Earlier quoted context omitted.
The latter. When "understand", "reason", "think", "feel", "believe", and any of a long list of similar words are in any title, it immediately makes me think the author already drank the kool aid.
I agree with “feel” and “believe” but what words would you suggest instead of “understand” and “reason’?
None. Don't anthropomorphize at all. Note that "understanding" has now been removed from the HN title but not the linked pdf.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#46Earlier quoted context omitted.
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
I keep wondering whether people have actually examined how this work draws its conclusions before citing it. This is science at its worst, where you start at an inflammatory conclusion and work backwards. There is nothing particularly novel presented here, especially not in the mathematics; obviously performance will degrade on out-of-distribution tasks (and will do so for humans under the same formulation), but the…
"This is science at its worst, where you start at an inflammatory conclusion and work backwards"
Science starts with a guess and you run experiments to test.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#47I have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?
Does it matter from a practical point of view? It's either true understanding or it's something else that's similar enough to share the same name.
The polygraph is a good example.
The "lie detector" is used to misguide people, the polygraph is used to measure autonomic arousal.
I think these misnomers can cause real issues like thinking the LLM is "reasoning".
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#48Earlier quoted context omitted.
I get that. But then if that option doesn't help, what I've seen is that the next followup is inevitably "have you tried doing/prompting x instead of y"
> I get that. But then if that option doesn't help, what I've seen is that the next followup is inevitably "have you tried doing/prompting x instead of y" Maybe I’m misunderstanding, but it sounds like you’re framing a completely normal proces (try, fail, adjust) as if it’s unreasonable? In reality, when something doesn’t work, it would seem to me that the obvious next step is to adapt and try again. This does not se…
I get trying and improving until you get it right. But I just can't make the bridge in my head around
1. this is magic and will one-shot your questions 2. but if it goes wrong, keep trying until it works
Plus, knowing it's all probabilistic, how do you know, without knowing ahead of time already, that the result is correct? Is that not the classic halting problem?
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#49Here's my experience: for some coding tasks where GPT 4.1, Claude Sonnet 4, Gemini 2.5 Pro were just spinning for hours and hours and getting nowhere, GPT 5 just did the job without a fuss. So, I switched immediately to GPT 5, and never looked back. Or at least I never looked back until I found out that my company has some Copilot limits for premium models and I blew through the limit. So now I keep my context small,…
Interestingly I'm experiencing the opposite as you. Was mostly using Claude Sonnet 4 and GPT 4.1 through copilot for a few months and was overall fairly satisfied with it. First task I threw at GPT 5, it excelled in a fraction of the time Sonnet 4 normally takes, but after a few iterations, it all went downhill. GPT 5 almost systematically does things I didn't ask it to do. After failing to solve an issue for almost an hour, I switched back to Claude which fixed it in the first try. YMMV
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#50Earlier quoted context omitted.
I keep wondering whether people have actually examined how this work draws its conclusions before citing it. This is science at its worst, where you start at an inflammatory conclusion and work backwards. There is nothing particularly novel presented here, especially not in the mathematics; obviously performance will degrade on out-of-distribution tasks (and will do so for humans under the same formulation), but the…
"This is science at its worst, where you start at an inflammatory conclusion and work backwards" Science starts with a guess and you run experiments to test.
True, but the experiments are engineered to give results they want. It's a mathematical certainty that the performance will drop off here, but is not an accurate assessment of what is going on at scale. If you present an appropriately large and well-trained model with in-context patterns, it often does a decent job, even when it isn't trained on them. By nerfing the model (4 layers), the conclusion is foregone.
I honestly wish this paper actually showed what it claims, since it is a significant open problem to understand CoT reasoning relative to the underlying training set.