Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

Post reply on HN
Print view View original

291–300 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#291

Post by xarope »

TurdF3rguson wrote: post #278
n8henrie wrote: post #252

Earlier quoted context omitted.

Unfortunately the training data is absolute garbage. Diagnostic standards in (at least emergency, but I think other specialties) medicine are largely a joke -- ultimately it's often either autopsy or "expert consensus." We get to bill more for more serious diagnoses. The amount of patients I see with a "stroke" or "heart attack" diagnosis that clearly had no such thing is truly wild. We can be sued for tens of millio…

You just get 1M doctors to wear body cams for a year. Now you have a model that has thousands of times your experience with patients, encyclopedic knowledge of every ailment including ones that never present in your geography, read all the latest papers, etc.. I don't understand how you think this doesn't win vs a human doctor.

In healthcare, HIPAA/GDPR equivalent would block this. Let's be realistic in our discussion; this is not the same as google buying up a library worth of books, scanning and destroying them

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#292

Post by melagonster »

YetAnotherNick wrote: post #280
teleforce wrote: post #265

Earlier quoted context omitted.

>What is the specific capability (or combination of capabilities) that people believe will remain permanently (or at least for decades) where a top medical AI cannot match or exceed the performance of a good human doctor? Let's put liability and ethics aside, let's be purely objective about it. You cannot simply put liability and ethics aside, after all there's Hippocatic oath that's fundamental to the practice physi…

Assume if you know for certain that AI has better senstivity and specificity than your local physician for the particular diagnosis, which likely would be the case now or in few years. Would you purposefully get inferior consultation just because of Hippocatic oath?

Doctors will apply AI sooner than patient, and they can check these results with confidence.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#293

Post by 01100011 »

guidedlight wrote: post #270
01100011 wrote: post #134

I wouldn't put much weight in this study, but I think a lot of us can still attest to the usefulness of LLMs in self-diagnostics. The reality in the US is that it is difficult to get the attention and care of a doctor so we're left having to do it ourselves. 10 years ago you'd hear docs complaining about patients coming in with things they found on google but now I don't think there's an alternative. Case in point, I…

I agree. I think the issue with LLM’s are not with the correct diagnoses’s but rather the incorrect ones. Real doctors tend to have a degree of cautiousness. I would rather a real doctor be hesitate and seek more information, than an alarmist LLM suggesting I have cancer.

Yeah apparently my comment wasn't clear enough. If you can get the opinion of a doctor then good for you. I'm saying an LLM is the best some of us can get.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#294

Post by lpcvoid »

tehjoker wrote: post #230
analog31 wrote: post #161

Earlier quoted context omitted.

I used to think this too. But the past couple of years have soured my taste for "dismantle and replace" of vital institutions. I still think healthcare needs to be reformed, and I hope that insurance will someday be a thing of a past, but I've hung up my chain saw for now.

You've witnessed a dismantle and replace effort by the right wing that wishes to squeeze everything to make rich people more money. An effort by the left would destroy the private insurance scheme and build up medicare. Completely different and you'd get something functional. When the wrong targets get destroyed, everyone suffers. When parasitic forces are destroyed, the system functions better. It's the difference b…

Why is this downvoted? What the (far) right wing has done to american institutions is incredibly destructive and it will take decades to fix it, if that's even possible. People are hurting because of it.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#295

Post by ivolimmen »

I can't help to visualize the scene in Idiocracy where there is an examination. The guy gets multiple wires that gets put in his hands, mouth and rectum. The guy that assists (aka the doctor) switches the wires after each person.

If we trust machines to much...

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#296

Post by SkiFire13 »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

> we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors), if we already have this assumption for software engineers

You first have to assume this for software engineers. Not everyone agree with that (note: that doesn't mean the same people don't agree that AI is not _useful_).

AIs still have a ton of issues that would be devastating in a doctor. Remember all the AIs mistakingly deleting production DBs? Now imagine they prescribed a medicine cocktail that killed the patient instead. No thanks. There's a totally different bar to the consequences of mistakes.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#297

Post by arcfour »

Intralexical wrote: post #253
Aurornis wrote: post #132

Earlier quoted context omitted.

When you read through the article it shows that the gap between doctors and LLMs actually disappeared (in terms of statistical significance) once both were allowed to read the full case notes. The headline is quoting a number based on guessed diagnoses from nurse's notes. The LLM was happier to take guesses from the selected case studies than the doctors is my guess.

Not only is the study testing something which only vaguely resembles how doctors diagnose patients, but isolated accuracy percentages are also a terrible way to measure healthcare quality. If 90% of patients have a cold, and 10% have metastatic aneuristic super-boneitis, then you can get 90% accuracy by saying every patient has a cold. I would expect a probabilistic token-prediction machine to be good at that. But ho…

What percentage of patients have blood clots in their lungs and a history of lupus, like the article described? That's not on the same level as a common cold at all.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#298

Post by drawfloat »

whiplash451 wrote: post #284
teleforce wrote: post #265

Earlier quoted context omitted.

>What is the specific capability (or combination of capabilities) that people believe will remain permanently (or at least for decades) where a top medical AI cannot match or exceed the performance of a good human doctor? Let's put liability and ethics aside, let's be purely objective about it. You cannot simply put liability and ethics aside, after all there's Hippocatic oath that's fundamental to the practice physi…

What do imperfect, biased and expensive human doctors add to the « liability and ethics » question exactly?

Human judgement and accountability

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#299

Post by bluegatty »

teleforce wrote: post #265
pixel_popping wrote: post #25

Earlier quoted context omitted.

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

>What is the specific capability (or combination of capabilities) that people believe will remain permanently (or at least for decades) where a top medical AI cannot match or exceed the performance of a good human doctor? Let's put liability and ethics aside, let's be purely objective about it. You cannot simply put liability and ethics aside, after all there's Hippocatic oath that's fundamental to the practice physi…

I think this is mixing streams here.

Try narrowing the scope to remove the word 'AI' and just think 'Blood Test'.

We accept that machines can do these things faster and better than humans, and we don't lose sleep over it.

The AI will be faster and better than humans at so many things, obviously.

"Hipprocatic Oath" isn't hugely relevant to diagnosis etc.

These are systems we are measuring, that's it.

Obviously - treatment and other things, we'll need 'Hipprocatic Humans' ... but most of this is Engineering.

I don't think doctors will even trust their own judgment for many things for very long, their role will evolve as it has for a long time.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#300

Post by theshrike79 »

mawadev wrote: post #71
theshrike79 wrote: post #28

I'll repeat my idea on how this MUST be done: 1. AI gets data about the patient and makes a diagnosis. This is NOT shown to doctor yet. 2. Doctor does their stuff, writes down their diagnosis. This diagnosis is locked down and versioned. 3. Doctor sees AI's diagnosis 4. Doctor can adjust their diagnosis, BUT the original stays in the system. This way the AI stays as the assistant and won't affect the doctor's decisio…

This still promotes metacognitive laziness later down the road as the doctor can hand in something quickly and rely on AI to close that gap.

The magic is in the initial diagnosis being written down, saved and locked.

It's trivial to analyse the pre/post AI involvement doctor diagnosis manually and see what's going on.

If a doctor is just putting "asdljasdaskjd" on the initial to unlock the AI answer, they should be promptly fired.

Post reply on HN