so since reasoning_effort is not discussed anywhere, I assume you used the default which is "medium"?
From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
51–60 of 102 posts
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#52Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#53Here's my experience: for some coding tasks where GPT 4.1, Claude Sonnet 4, Gemini 2.5 Pro were just spinning for hours and hours and getting nowhere, GPT 5 just did the job without a fuss. So, I switched immediately to GPT 5, and never looked back. Or at least I never looked back until I found out that my company has some Copilot limits for premium models and I blew through the limit. So now I keep my context small,…
Interestingly I'm experiencing the opposite as you. Was mostly using Claude Sonnet 4 and GPT 4.1 through copilot for a few months and was overall fairly satisfied with it. First task I threw at GPT 5, it excelled in a fraction of the time Sonnet 4 normally takes, but after a few iterations, it all went downhill. GPT 5 almost systematically does things I didn't ask it to do. After failing to solve an issue for almost…
Yeah, GPT 5 got into death loops faster than any other LLM, and I stopped using it for anything more than UI prototypes.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#54[deleted]
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#55so since reasoning_effort is not discussed anywhere, I assume you used the default which is "medium"?
Also, were tool calls allowed? The point of reasoning models is to delete the facts so finite capacity goes towards the dense reasoning engine rather than recall, with the facts sitting elsewhere.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#56Feels like a mixed bag vs regression? eg - GPT-5 beats GPT-4 on factual recall + reasoning (HeadQA, Medbullets, MedCalc). But then slips on structured queries (EHRSQL), fairness (RaceBias), evidence QA (PubMedQA). Hallucination resistance better but only modestly. Latency seems uneven (maybe more testing?) faster on long tasks, slower on short ones.
GPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.
I wonder how that math works out. GPT-5 keeps triggering a thinking flow even for relatively simple queries, so each token must be a magnitude cheaper to make this worth the trade-off in performance.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#57Earlier quoted context omitted.
I agree with “feel” and “believe” but what words would you suggest instead of “understand” and “reason’?
None. Don't anthropomorphize at all. Note that "understanding" has now been removed from the HN title but not the linked pdf.
Why not? We are trying to evaluate AI's capabilities. It's OBVIOUS that we should compare it to our only prior example of intelligence -- humans. Saying we shouldn't compare or anthropomorphize machine is a ridiculous hill to die on.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#58Earlier quoted context omitted.
This is not a good analogy because reasoning models are not choosing the best from a set of attempts based on knowledge of the correct answer. It really is more like what it sounds like: “did you think about it longer until you ruled out various doubts and became more confident?” Of course nobody knows quite why directing more computation in this way makes them better, and nobody seems to take the reasoning trace too…
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
The other commenter is more articulate, but you simply cannot draw the conclusion from this paper that reasoning models don't work well. They trained tiny little models and showed they don't work. Big surprise! Meanwhile every other piece of evidence available shows that reasoning models are more reliable at sophisticated problems. Just a few examples.
- https://arcprize.org/leaderboard
- https://aider.chat/docs/leaderboards/
- https://arstechnica.com/ai/2025/07/google-deepmind-earns-gol...
Surely the IMO problems weren't "within the bounds" of Gemini's training data.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#59GPT-5 is like an autistic savant
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#60Earlier quoted context omitted.
"This is science at its worst, where you start at an inflammatory conclusion and work backwards" Science starts with a guess and you run experiments to test.
True, but the experiments are engineered to give results they want. It's a mathematical certainty that the performance will drop off here, but is not an accurate assessment of what is going on at scale. If you present an appropriately large and well-trained model with in-context patterns, it often does a decent job, even when it isn't trained on them. By nerfing the model (4 layers), the conclusion is foregone. I hon…
Without a provable hold out, claim that "large models do fine on unseen patterns" is unfalsifiable. In controlled from scratch training, CoT performance collapses under modest distribution shift, even with plausible chains. If you have results where the transformation family is provably excluded from training and a large model still shows robust CoT, please share them. Otherwise this paper’s claim stands for the regime it tests.