Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

Post reply on HN
Print view View original

361–370 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#361

Post by ricardobayes »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

It's having a general understanding/view of the "baseline", aka healthy anatomy. This is something LLMs will never have, that's why never have true reasoning, for the lack of "worldview" and they never know if they are hallucinating. To aid doctors, we don't need LLMs but rather, computer vision, pattern recognition as you correctly point out.

But it's important not to rely on it. Doctors can easily recognize and correct measurements with incorrect input, e.g. ECG electrodes being used in reverse order.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#362

Post by mrpopo »

scotty79 wrote: post #356
miki123211 wrote: post #354

Earlier quoted context omitted.

> What is the specific capability (or combination of capabilities) The ability to go to prison / be stripped of a license when something goes wrong. A single doctor will care for far fewer patients in their career than an AI system will. Even if the AI system is 10x less likely to make mistakes, the sheer number of patients will make it much more likely to make a mistake somewhere. With a single doctor, the PR and le…

> > What is the specific capability (or combination of capabilities) > The ability to go to prison / be stripped of a license when something goes wrong. So basically you need a person to blame if things don't go the best way possible?

How else do you guarantee that things will keep going the best way possible in the future? The magical hand of the market?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#363

Post by ricardobayes »

scotty79 wrote: post #356
miki123211 wrote: post #354

Earlier quoted context omitted.

> What is the specific capability (or combination of capabilities) The ability to go to prison / be stripped of a license when something goes wrong. A single doctor will care for far fewer patients in their career than an AI system will. Even if the AI system is 10x less likely to make mistakes, the sheer number of patients will make it much more likely to make a mistake somewhere. With a single doctor, the PR and le…

> > What is the specific capability (or combination of capabilities) > The ability to go to prison / be stripped of a license when something goes wrong. So basically you need a person to blame if things don't go the best way possible?

No, but someone needs to bear responsibility. Whether that's a doctor, or a CEO directly, ordering the replacement of a radiologist by AI. If things go sideways, there needs to be a chain or responsibility.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#364

Post by somethingsome »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

95% of the cases are easy for both doctors and AI, where doctors excel are the difficult cases where there is only a very limited amount of training data ;) something AI is not yet ready to handle at all.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#365

Post by guidedlight »

01100011 wrote: post #293
guidedlight wrote: post #270

Earlier quoted context omitted.

I agree. I think the issue with LLM’s are not with the correct diagnoses’s but rather the incorrect ones. Real doctors tend to have a degree of cautiousness. I would rather a real doctor be hesitate and seek more information, than an alarmist LLM suggesting I have cancer.

Yeah apparently my comment wasn't clear enough. If you can get the opinion of a doctor then good for you. I'm saying an LLM is the best some of us can get.

Oh right. Almost everyone in the world has free and easy access to actual doctors.

For that one country that doesn’t maybe universal healthcare can be an Anthropic model.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#366

Post by epmaybe »

I’m in ophthalmology where AI diagnostics have been promised for almost a decade. We have FDA approved diagnostics for diabetic retinopathy screening that has been commercially available since 2018, and papers claiming board certified ophthalmologist level classification accuracy as far back as inceptionv3. Maybe it’s just an economic barrier but these tools still haven’t made any meaningful impact in the US. Other countries without healthcare access? It’s helpful for culling the herd, but it doesn’t fix the last mile problem of what you do when you find referable disease that needs treatment.

My philosophical take: if AI can outperform the average, it’s probably a net benefit for society that I won’t have a job. Until then, I’m going to take my income and save up for an early retirement.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#367

Post by HPsquared »

somethingsome wrote: post #364
pixel_popping wrote: post #25

Earlier quoted context omitted.

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

95% of the cases are easy for both doctors and AI, where doctors excel are the difficult cases where there is only a very limited amount of training data ;) something AI is not yet ready to handle at all.

To safely handle those difficult cases, you need an AI that can reliably say "I don't know".

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#368

Post by HPsquared »

analog31 wrote: post #161
ianbutler wrote: post #111

Earlier quoted context omitted.

Sounds like we need to dismantle and replace this broadly dysfunctional system at multiple points. It's not like the US insurance landscape is anywhere close to the best way of handling healthcare if you look at many places in the world.

I used to think this too. But the past couple of years have soured my taste for "dismantle and replace" of vital institutions. I still think healthcare needs to be reformed, and I hope that insurance will someday be a thing of a past, but I've hung up my chain saw for now.

Replace first, then the old broken one will fade away.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#369

Post by xbmcuser »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

If all the curated data is really shared with an AI over time they will be better than most individual doctors. I personally think AI could be a great triage system.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#370

Post by notahacker »

TurdF3rguson wrote: post #343
xarope wrote: post #291

Earlier quoted context omitted.

In healthcare, HIPAA/GDPR equivalent would block this. Let's be realistic in our discussion; this is not the same as google buying up a library worth of books, scanning and destroying them

There are other countries, and the patients in them all have similar data

Other countries actually don't necessarily have a similar mix of ailments, median patient appearance and style of communication or even recommended course of action and most of the ones with more sophisticated medical care also have strict medical privacy laws. If you're genuinely unaware of this, I'm not sure you're in a position to be making "one year with a camera, how hard can it be" arguments...

(Where AI is likely to actually excel in medicine is parsing datasets that are much easier to do context free number crunching on than ER rooms, some of which physicians don't even have access to ...)

Post reply on HN