Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

Post reply on HN
Print view View original

71–80 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#71

Post by mawadev »

theshrike79 wrote: post #28

I'll repeat my idea on how this MUST be done: 1. AI gets data about the patient and makes a diagnosis. This is NOT shown to doctor yet. 2. Doctor does their stuff, writes down their diagnosis. This diagnosis is locked down and versioned. 3. Doctor sees AI's diagnosis 4. Doctor can adjust their diagnosis, BUT the original stays in the system. This way the AI stays as the assistant and won't affect the doctor's decisio…

This still promotes metacognitive laziness later down the road as the doctor can hand in something quickly and rely on AI to close that gap.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#72

Post by cyberax »

OptionOfT wrote: post #45

As a 37 year old male with 2 THRs I'm glad the AI was NOT used in my diagnosis. All the models that I used to look at my x-rays said nothing was wrong, even when adding symptoms. When adding age it said the patient was too young. (I was ~3 months away from wheelchair bound in those x-rays). The worst one was Gemini. Upload an x-ray of just the right hip, and it started to talk about how good the left hip looked like.…

The general AI models are useless if you need precision. They are designed to create/analyze pretty pictures.

But specialized models can be inhumanly good. I know, our main product is a model that does _precise_ analysis :)

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#73

Post by ForceBru »

gherkinnn wrote: post #35
pixel_popping wrote: post #25

Earlier quoted context omitted.

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

"Human problems can't be solved with technology" is just wrong, unless you have narrower definitions of a "human problem" or "technology".

For instance, transportation is a "human problem". It's being successfully solved with such technologies as cars, trains, planes, etc. Growing food at scale is a "human problem" that's being successfully solved by automation. Computing... stuff could be a "human problem" too. It's being successfully solved by computers. If "human problems" are more psychological, then again, you can use the Internet to keep in touch with people, so again technology trying to solve a human problem.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#74

Post by mawadev »

troupo wrote: post #65
theshrike79 wrote: post #28

I'll repeat my idea on how this MUST be done: 1. AI gets data about the patient and makes a diagnosis. This is NOT shown to doctor yet. 2. Doctor does their stuff, writes down their diagnosis. This diagnosis is locked down and versioned. 3. Doctor sees AI's diagnosis 4. Doctor can adjust their diagnosis, BUT the original stays in the system. This way the AI stays as the assistant and won't affect the doctor's decisio…

5. Doctors delegate everything to AI assistants because humans are lazy, especially if those AI assistants are correct some significant portion of the time

Then the claim may be that you don't need that many doctors anymore and that one doctor can do the job of X doctors in less time which has the economical effect that there is less demand for/supply of doctors, which then results in a home grown shortage of doctors, since less people are incentivized to become doctors...

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#75

Post by mhitza »

gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I think AI can be useful in any kind of context interpretation, but not make a decision.

Could be running in the background on patient data and message the doctor "I see X in the diagnostic, have you ruled out Y, as it fits for reasons a, b, c?"

I like my coding agents the same way, inform me during review on things that I've missed. Instead of having me comb through what it generates on a first pass.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#76

Post by ForceBru »

rowanG077 wrote: post #54
gherkinnn wrote: post #35

Earlier quoted context omitted.

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

I would personally vastly, vastly prefer to go to a robot doctor, who diagnoses, treats and nurses me. What exactly do I need from a human here? Except of course being the one making the system.

Emotional support. Some human doctors absolutely radiate confidence and a kind of "you're gonna be okay" attitude. For me, this helps a lot. I'm not sure a machine can do this.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#78

Post by Chinjut »

mday27 wrote: post #46
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

hallucination on steroids, wow. I had to read through the abstract to believe it: "In the most extreme case, our model achieved the top rank on a standard chest Xray question-answering benchmark without access to any images."

I still don't quite understand, after skimming the paper. How does it achieve high scores without access to the images (beating even humans with access to the images)?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#79

Post by jmcgough »

LLMs can be a useful second opinion for a highly educated patient with good insight into their health and body, but this is not the average patient I see in an urban emergency department. Many patients can't give a cohesive history without a skilled clinician who can ask the right questions and read between the lines.

I am very skeptical of studies like this that don't adequately reflect real world conditions, but when I was a software engineer I probably wouldn't have understood what "real" medicine is like either.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#80

Post by dyauspitr »

gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I think the bigger takeaway here is that 50% of the time doctors will miss what you have.
Post reply on HN