Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

fertrevino.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#81

Post by UltraSane »

TrainedMonkey wrote: post #22
aresant wrote: post #11

Feels like a mixed bag vs regression? eg - GPT-5 beats GPT-4 on factual recall + reasoning (HeadQA, Medbullets, MedCalc). But then slips on structured queries (EHRSQL), fairness (RaceBias), evidence QA (PubMedQA). Hallucination resistance better but only modestly. Latency seems uneven (maybe more testing?) faster on long tasks, slower on short ones.

GPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.

Since the routing is opaque they can dynamically route queries to cheaper models when demand is high.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#82

Post by insin »

vexna wrote: post #78
sema4hacker wrote: post #33

Earlier quoted context omitted.

The latter. When "understand", "reason", "think", "feel", "believe", and any of a long list of similar words are in any title, it immediately makes me think the author already drank the kool aid.

kool aid or not -- "reasoning" is already part of the LLM verbiage (e.g `reasoning` models having `reasoningBudget`). The meaning might not be 1:1 to human reasoning, but when the LLM shows its "reasoning" it does look _appear_ like a train of thought. If I had to give what it's doing a name (like I'm naming a function), I'd be hard pressed to not go with something like `reason`/`think`.

    prefillContext()

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#83

Post by oezi »

username135 wrote: post #10

I wonder what changed with the models that created regression?

There is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#84

Post by yzydserd »

rbinv wrote: post #80
narrator wrote: post #24

Earlier quoted context omitted.

I agree. I have found GPT-5 significantly worse on medical queries. It feels like it skips important details and is much worse than o3, IMHO. I have heard good things about GPT-5 Pro, but that's not cheap. I wonder if part of the degraded performance is where they think you're going into a dangerous area and they get more and more vague, for example like they demoed on launch day with the fireworks example. It gets v…

Afaik, there is currently no "GPT-5 Pro". Did you mean o3-pro or o1-pro (via API)? Currently, GPT-5 sits at $10/1M output tokens, o3-pro at $80, and o1-pro at a whopping $600: https://platform.openai.com/docs/pricing Of course this is not indicative of actual performance or quality per $ spent, but according to my own testing, their performance does seem to scale in line with their cost.

O5-pro is available through the ChatGPT UI with a “Pro” plan. I understand that like o3 pro it is a high compute large context invocation of underlying models.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#85

Post by rbinv »

yzydserd wrote: post #84
rbinv wrote: post #80

Earlier quoted context omitted.

Afaik, there is currently no "GPT-5 Pro". Did you mean o3-pro or o1-pro (via API)? Currently, GPT-5 sits at $10/1M output tokens, o3-pro at $80, and o1-pro at a whopping $600: https://platform.openai.com/docs/pricing Of course this is not indicative of actual performance or quality per $ spent, but according to my own testing, their performance does seem to scale in line with their cost.

O5-pro is available through the ChatGPT UI with a “Pro” plan. I understand that like o3 pro it is a high compute large context invocation of underlying models.

Thanks, I was not aware! I thought they offered all their models via their API.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#86

Post by mastercheif »

rbinv wrote: post #80
narrator wrote: post #24

Earlier quoted context omitted.

I agree. I have found GPT-5 significantly worse on medical queries. It feels like it skips important details and is much worse than o3, IMHO. I have heard good things about GPT-5 Pro, but that's not cheap. I wonder if part of the degraded performance is where they think you're going into a dangerous area and they get more and more vague, for example like they demoed on launch day with the fireworks example. It gets v…

Afaik, there is currently no "GPT-5 Pro". Did you mean o3-pro or o1-pro (via API)? Currently, GPT-5 sits at $10/1M output tokens, o3-pro at $80, and o1-pro at a whopping $600: https://platform.openai.com/docs/pricing Of course this is not indicative of actual performance or quality per $ spent, but according to my own testing, their performance does seem to scale in line with their cost.

GPT-5 Pro is only available on ChatGPT with a ChatGPT Pro subscription.

Supposedly it fires off multiple parallel thinking chains and then essentially debates with itself to net a final answer.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#87

Post by 0xDEAFBEAD »

So which of these benchmarks are most relevant for an ordinary user who wants to talk to AI about their health issues?

I'm guessing HeadQA, Medbullets, MedHallu, and perhaps PubMedQA? (Seems to me that "unsupported speculation" could be a good thing for a patient who has yet to receive a diagnosis...)

Maybe in practice it's better to look at RAG benchmarks, since a lot of AI tools will search online for information before giving you an answer anyways? (Memorization of info would matter less in that scenario)

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#88

Post by slashdev »

JimDabell wrote: post #65
TrainedMonkey wrote: post #22

Earlier quoted context omitted.

GPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.

I’ve found that it’s super likely to get stuck repeating the exact same incorrect response over and over. It used to happen occasionally with older models, but it happens frequently now. Things like: Me: Is this thing you claim documented? Where in the documentation does it say this? GPT: Here’s a long-winded assertion that what I said before was correct, plus a link to an unofficial source that doesn’t back me up. M…

If one conversation goes in a bad direction, it's often best to just start over. The bad context often poisons the existing session.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#89

Post by Workaccount2 »

robrenaud wrote: post #77
dcre wrote: post #58

Earlier quoted context omitted.

The other commenter is more articulate, but you simply cannot draw the conclusion from this paper that reasoning models don't work well. They trained tiny little models and showed they don't work. Big surprise! Meanwhile every other piece of evidence available shows that reasoning models are more reliable at sophisticated problems. Just a few examples. - https://arcprize.org/leaderboard - https://aider.chat/docs/lead…

The Gemini IMO result used a specifically fine tuned model for math. Certainly they weren't training on the unreleased problems. Defining out of distribution gets tricky.

Every human taking that exam has fine tuned for math, specifically on IMO problems.

Re: From GPT-4 to GPT-5: Measuring progress through MedHELM [pdf]

#90

Post by Workaccount2 »

hodgehog11 wrote: post #37
kumarvvr wrote: post #30

I have an issue with the words "understanding", "reasoning", etc when talking about LLMs. Are they really understanding, or putting out a stream of probabilities?

What does understanding mean? Is there a sensible model for it? If not, we can only judge in the same way that we judge humans: by conducting examinations and determining whether the correct conclusions were reached. Probabilities have nothing to do with it; by any appropriate definition, there exist statistical models that exhibit "understanding" and "reasoning".

https://ai.vixra.org/pdf/2506.0065v1.pdf

Lays out pretty well what our current knowledge on understanding is

Post reply on HN