Earlier quoted context omitted.
Not sure but with each release it feels like they’re just wiping the dirt around and not actually cleaning.
Obligxkcd: https://xkcd.com/1838/
loved the cartoon :)
From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
71–80 of 102 posts
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#72Earlier quoted context omitted.
This is not a good analogy because reasoning models are not choosing the best from a set of attempts based on knowledge of the correct answer. It really is more like what it sounds like: “did you think about it longer until you ruled out various doubts and became more confident?” Of course nobody knows quite why directing more computation in this way makes them better, and nobody seems to take the reasoning trace too…
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
This is not the slam dunk you think it is. Thinking longer genuinely provides better accuracy. Sure there are decreasing returns to increasing thinking tokens.
GPT 5 fast gets many things wrong but switching to the thinking model fixes the issues very often.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#73Earlier quoted context omitted.
This is not a good analogy because reasoning models are not choosing the best from a set of attempts based on knowledge of the correct answer. It really is more like what it sounds like: “did you think about it longer until you ruled out various doubts and became more confident?” Of course nobody knows quite why directing more computation in this way makes them better, and nobody seems to take the reasoning trace too…
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
A viable consideration is that the models will hone in on and reinforce an incorrect answer - a natural side effect of the LLM technology wanting to push certain answers higher in probability and repeat anything in context.
Regardless of being in conversation or thinking context this doesn't prevent the model from speaking the wrong answer so the paper on the illusion of thinking makes sense.
What actually seems to be happening is a form of conversational prompting. Of course with the right conversation back and forth with an LLM you can inject knowledge in a way that causes the natural distribution to shift (again - side effect of the LLM tech.) but by itself it won't naturally get the answer perfect every time.
If this extended thinking were actually working you would expect the LLM to be able to logically conclude an answer with very high accuracy 100% of the time which it does not.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#74Earlier quoted context omitted.
True, but the experiments are engineered to give results they want. It's a mathematical certainty that the performance will drop off here, but is not an accurate assessment of what is going on at scale. If you present an appropriately large and well-trained model with in-context patterns, it often does a decent job, even when it isn't trained on them. By nerfing the model (4 layers), the conclusion is foregone. I hon…
Without a provable hold out, claim that "large models do fine on unseen patterns" is unfalsifiable. In controlled from scratch training, CoT performance collapses under modest distribution shift, even with plausible chains. If you have results where the transformation family is provably excluded from training and a large model still shows robust CoT, please share them. Otherwise this paper’s claim stands for the regi…
I don't buy this for the simple fact that benchmarks show much better performance on thinking than on non thinking models. Benchmarks already consider the generalisation and "unseen patterns" aspect.
What would be your argument against
1. COT models performing way better in benchmarks than normal models
2. people choose to use the COT models in day to day life because they actually find that it gives better performance
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#75Earlier quoted context omitted.
I’ve found that it’s super likely to get stuck repeating the exact same incorrect response over and over. It used to happen occasionally with older models, but it happens frequently now. Things like: Me: Is this thing you claim documented? Where in the documentation does it say this? GPT: Here’s a long-winded assertion that what I said before was correct, plus a link to an unofficial source that doesn’t back me up. M…
Go back and edit a prompt of yours in the conversation instead of continuing with garbage in the context.
That's a good tip, I didn't know you could do that.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#76Here's my experience: for some coding tasks where GPT 4.1, Claude Sonnet 4, Gemini 2.5 Pro were just spinning for hours and hours and getting nowhere, GPT 5 just did the job without a fuss. So, I switched immediately to GPT 5, and never looked back. Or at least I never looked back until I found out that my company has some Copilot limits for premium models and I blew through the limit. So now I keep my context small,…
any thread on HN about AI (there's constantly at least one in homepage nowadays) goes like this:
"in my experience [x model] one shots everything and [y model] stumbles and fumbles like a drunkard", for _any_ combination of X and Y.
I get the idea of sharing what's working and what's not, but at this point it's clear that there are more factors to using these with success and it's hard to replicate other people's successful workflows.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#77Earlier quoted context omitted.
Bad news: it doesn't seem to work as well as you might think: https://arxiv.org/pdf/2508.01191 As one might expect, because the AI isn't actually thinking, it's just spending more tokens on the problem. This sometimes leads to the desired outcome but the phenomenon is very brittle and disappears when the AI is pushed outside the bounds of its training. To quote their discussion, "CoT is not a mechanism for genuine lo…
The other commenter is more articulate, but you simply cannot draw the conclusion from this paper that reasoning models don't work well. They trained tiny little models and showed they don't work. Big surprise! Meanwhile every other piece of evidence available shows that reasoning models are more reliable at sophisticated problems. Just a few examples. - https://arcprize.org/leaderboard - https://aider.chat/docs/lead…
The Gemini IMO result used a specifically fine tuned model for math.
Certainly they weren't training on the unreleased problems. Defining out of distribution gets tricky.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#78I have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?
The latter. When "understand", "reason", "think", "feel", "believe", and any of a long list of similar words are in any title, it immediately makes me think the author already drank the kool aid.
kool aid or not -- "reasoning" is already part of the LLM verbiage (e.g `reasoning` models having `reasoningBudget`).
The meaning might not be 1:1 to human reasoning, but when the LLM shows its "reasoning" it does look _appear_ like a train of thought. If I had to give what it's doing a name (like I'm naming a function), I'd be hard pressed to not go with something like `reason`/`think`.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#79Earlier quoted context omitted.
Does it matter from a practical point of view? It's either true understanding or it's something else that's similar enough to share the same name.
The polygraph is a good example. The "lie detector" is used to misguide people, the polygraph is used to measure autonomic arousal. I think these misnomers can cause real issues like thinking the LLM is "reasoning".
Agreed, but in the case of the lie detector, it seems it's a matter of interpretation. In the case of LLMs, what is it? Is it a matter of saying "It's a next-word calculator that uses stats, matrices and vectors to predict output" instead of "Reasoning simulation made using a neural network"? Is there a better name? I'd say it's "A static neural network that outputs a stream of words after having consumed textual input, and that can be used to simulate, with a high level of accuracy, the internal monologue of a person who would be thinking about and reasoning on the input". Whatever it is, it's not reasoning, but it's not a parrot either.
Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]
#80Earlier quoted context omitted.
GPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.
I agree. I have found GPT-5 significantly worse on medical queries. It feels like it skips important details and is much worse than o3, IMHO. I have heard good things about GPT-5 Pro, but that's not cheap. I wonder if part of the degraded performance is where they think you're going into a dangerous area and they get more and more vague, for example like they demoed on launch day with the fireworks example. It gets v…
Afaik, there is currently no "GPT-5 Pro". Did you mean o3-pro or o1-pro (via API)?
Currently, GPT-5 sits at $10/1M output tokens, o3-pro at $80, and o1-pro at a whopping $600: https://platform.openai.com/docs/pricing
Of course this is not indicative of actual performance or quality per $ spent, but according to my own testing, their performance does seem to scale in line with their cost.