91–100 of 102 posts
I wonder what changed with the models that created regression?
There is some speculation that GPT-5 uses a router to decide which expert model to deploy (e.g. to mini vs o/thinking models). So the router might decide that the query can be solved by a cheaper model and this model gives worse results.
Earlier quoted context omitted.
The other commenter is more articulate, but you simply cannot draw the conclusion from this paper that reasoning models don't work well. They trained tiny little models and showed they don't work. Big surprise! Meanwhile every other piece of evidence available shows that reasoning models are more reliable at sophisticated problems. Just a few examples. - https://arcprize.org/leaderboard - https://aider.chat/docs/lead…
The Gemini IMO result used a specifically fine tuned model for math. Certainly they weren't training on the unreleased problems. Defining out of distribution gets tricky.
This is false.
https://x.com/YiTayML/status/1947350087941951596
This is false even for the OpenAI model
https://x.com/polynoamial/status/1946478250974200272
"Typically for these AI results, like in Go/Dota/Poker/Diplomacy, researchers spend years making an AI that masters one narrow domain and does little else. But this isn’t an IMO-specific model. It’s a reasoning LLM that incorporates new experimental general-purpose techniques."
Earlier quoted context omitted.
GPT-5 feels like cost engineering. The model is incrementally better, but they are optimizing for least amount of compute. I am guessing investors love that.
I’ve found that it’s super likely to get stuck repeating the exact same incorrect response over and over. It used to happen occasionally with older models, but it happens frequently now. Things like: Me: Is this thing you claim documented? Where in the documentation does it say this? GPT: Here’s a long-winded assertion that what I said before was correct, plus a link to an unofficial source that doesn’t back me up. M…
Earlier quoted context omitted.
OK, we've removed all understanding from the title above.
Care to provide reasoning as to why?
The previous truncation ("From GPT-4 to GPT-5: Measuring Progress in Medical Language Understanding") was baity in the sense that the word 'understanding' was provoking objections and taking us down a generic tangent about whether LLMs really understand anything or not. Since that wasn't about the specific work (and since generic tangents are basically always less interesting*), it was a good idea to find an alternate truncation.
So I took out the bit that was snagging people ("understanding") and instead swapped in "MedHELM". Whatever that is, it's clearly something in the medical domain and has no sharp edge of offtopicness. Seemed fine, and it stopped the generic tangent from spreading further.
* https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Earlier quoted context omitted.
Do you yourself really understand, or are you just depolarizing neurons that have reached their threshold?
[flagged]
The idea is: if you have a substantive point, make it thoughtfully; if not, please don't comment until you do.
Earlier quoted context omitted.
Care to provide reasoning as to why?
The article's title was longer than 80 chars, which is HN's limit. There's more than one way to truncate it. The previous truncation ("From GPT-4 to GPT-5: Measuring Progress in Medical Language Understanding") was baity in the sense that the word 'understanding' was provoking objections and taking us down a generic tangent about whether LLMs really understand anything or not. Since that wasn't about the specific wor…
Generic Tangents is my new band's name.
Earlier quoted context omitted.
None. Don't anthropomorphize at all. Note that "understanding" has now been removed from the HN title but not the linked pdf.
Why not? We are trying to evaluate AI's capabilities. It's OBVIOUS that we should compare it to our only prior example of intelligence -- humans. Saying we shouldn't compare or anthropomorphize machine is a ridiculous hill to die on.
Feels like a mixed bag vs regression? eg - GPT-5 beats GPT-4 on factual recall + reasoning (HeadQA, Medbullets, MedCalc). But then slips on structured queries (EHRSQL), fairness (RaceBias), evidence QA (PubMedQA). Hallucination resistance better but only modestly. Latency seems uneven (maybe more testing?) faster on long tasks, slower on short ones.
Definitely seems like GPT5 is a very incremental improvement. Not what you’d expect if AGI were imminent.