Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

421–430 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#421
Hyped title. It was exclusively text-based diagnosis after physicians did the whole interview, exam, labs, etc.

Also, later in the encounter, with more chart information, AI scored 82%, physicians 70–79%; that difference was reportedly not statistically significant.

So current AI can aid in diagnosing like we've all known.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#422
post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

> if we already have this assumption for software engineers,

Assuming what exactly? That they write more code? Better code? Better designs? Better architecture?

Because only a few of the above assumptions are arghuably true.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#423

Earlier quoted context omitted.

One doctor didn't want to give me ritalin, so i went to another one. One was against it, the other one saw it as a good idea. I would love to have real data, real statistics etc.

[flagged]

Dude this relentless LLM optimism is exhausting

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#424
My spouse is an hematologist+oncologist. She and all of her coworkers use ChatGPT. Before then, they look stuff up on UpToDate [ https://www.uptodate.com/login ] (they sometimes still do). I went to medical school for three years and quit because I couldn't stand the rote memorization part of the studies. Too many facts to remember IMO.

Even as an AI-neutral person, I'm very confident that AI/ML based computer systems, once trained specifically for medicine, will consistently do better than human doctors because believe it or not, there are a lot of human errors made in medicine field (doctors just don't admit that and we don't know) due to lack of time by doctors or incompetence or simply forgetting a fact or two that they should have checked when diagnosing or coming up with a treatment.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#425

> "An AI and a pair of human doctors were each given the same standard electronic health record to read" This is handicapping the human doctors abilities. There is a lot more information a human doctor can gather even with a brief observation of the patient.

So o1 can do more with less?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#426
post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

I think it comes down to how much data we're comfortable feeding an AI. If the AI has cameras and/or microphones in the room and the patient is directly talking to the AI: I strongly suspect AIs will always achieve better outcomes than humans. However, this kind of configuration will be viewed very negatively in a medical context for the foreseeable future; outside of limited contexts like "let me take a picture of that mole"; and hobbling the AI to only a text input (or dictated text by the doctor) muddies the waters on who is performing better. There's a lot of intuition in the diagnosis of something like "the location of the pain aligns with appendicitis, but they just aren't in enough pain" that cannot come through in just the textual representation of what is happening; you need to hear the person's voice and see how they're holding their body. AI can do that, but will we let it do that?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#427
post #202
post #2

o1 is several generations old and was released in 2024. Is this some quite old research that took a long time to get published?

Medical research moves. Very. Slowly.

That's a good thing

The medical equivalent to "move fast and break things" would be "move fast and kill people"

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#428

Earlier quoted context omitted.

The bar for making ai useful is much lower though. It's enough to be better than nothing. Large populations also in the technically rich countries simply do not have access to a doctor. in Poland which has a free public Healthcare it takes literal years to get a single appointment sometimes.

Do you believe the issue is because they don't have enough technicians to diagnose or because they don't have enough x-ray machines? Or in a ER environment, how an AI would speed up things in a real way that improves patients' lives? We just minted the term "cognitive debt" for software engineers that cannot keep up with what the AI spits out. How would that apply to ER doctors, or any other kind of doctor?

I'm not talking in particular about the X rays. It's about general lack of hospitals, equipment and doctors.

In Europe, there are some rich cities which have on average one doctor per hundred people. And there are large areas in Eastern Europe that have ten times less than that.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#430
post #184

Earlier quoted context omitted.

I’m a normal weight, and get asked the same question. More importantly, I can tell them, “I have a regular cycle” and they WILL NOT take that as an answer. I HAVE to give them a date, and they will ask me to make one up if I can’t remember or want to decline giving them that information. Particularly given the alarming stories of people being prosecuted for having miscarriages, it feels ridiculous. If anything I hope…

> and they will ask me to make one up if I can’t remember or want to decline giving them that information Doesn't this suggest that they don't care what the answer is?

They, as an individual healthcare provider, don’t care. The system will not allow them to ignore it, though, so the system cares very much.
Post reply on HN