Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

Post reply on HN
Print view View original

151–160 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#151

Post by thaumasiotes »

fullstop wrote: post #123
ianbutler wrote: post #59

Earlier quoted context omitted.

So much of what I know from women in my life is that the human element of medicine is almost a strict negative for them. As a guy it hasn't been much better, but at least doctors listen to me when I say something.

Yes, yes, but when was your last period? This even translates to the pediatric space. I took all of my kids to the pediatrician because either they don't make comments to me like they do to my wife, or I don't take shit from them. I'm not sure which. Here's an example: My wife and daughter were there and the doctor asked what kind of milk my daughter was drinking. She said "whole milk" and the doctor made a comment a…

> My daughter is in her 20s now and is still small -- it's just the way she is. When she goes to see her primary, do you know what their first question is? "When was your last period."

Is that supposed to be a problem? How does it connect to the story in your comment?

The question seems to be warranted to me, since being underweight can stop you from menstruating. So if you find someone thin and her last period was off in the distant past, you can conclude that there's a problem and something should be done about it; if it was a couple of weeks ago, you can conclude that she's fine.

(It could also just be something that is automatically assessed as a potential indicator of all kinds of different things. Notably pregnancy. For me, it bothered me that whenever you have an appointment at Kaiser for any reason, part of their checkin procedure is asking you how tall you are. I'd answer, but eventually I started pointing out to them that I wasn't ever measuring my height and they were just getting the same answer from my memory over and over again. [By contrast, they also take your weight every time, but they do that by putting you on a scale and reading it off.] The fact that my height wasn't being remeasured didn't bother them; I'm not sure what that question is for.)

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#152

Post by adamtaylor_13 »

Despite what I suspect the general consensus on HN may be, this does not surprise me at all.

My wife was recently diagnosed with Mast Cell Activation Syndrome (MCAS) after a pretty scary series of ER visits. It's a very strange and stubborn autoimmune disease that manifests with a number of symptoms that, taken individually, could indicate damn near anything.

You could almost feel the doctors rolling their eyes as she explained her symptoms and medical history.

Anyway... it lit a bit of a fire in me to dig deeper, and one day Claude suggested MCAS. I started plugging in more labs, asking for Claude to cross-reference journals mentioning MCAS, and sure enough: it's MCAS.

idk what the moral of the story is except our current medical system is a joke. The doctors aren't the villains, but they sure aren't the heroes either.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#153

Post by thesmtsolver2 »

andai wrote: post #147
dragonwriter wrote: post #126

Earlier quoted context omitted.

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans No, I don’t see that we must. > if we already have this assumption for software engineers No, this doesn’t follow, and even if it did, while I am aware that the CEOs…

Self-improving system given enough time to self-improve doesn't beat non-self-improving system?

Why are human doctors non-self improving?

If anything, using AI, they may improve more than before.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#154

Post by NegativeK »

01100011 wrote: post #134

I wouldn't put much weight in this study, but I think a lot of us can still attest to the usefulness of LLMs in self-diagnostics. The reality in the US is that it is difficult to get the attention and care of a doctor so we're left having to do it ourselves. 10 years ago you'd hear docs complaining about patients coming in with things they found on google but now I don't think there's an alternative. Case in point, I…

I don't think that using LLMs for medicine is an appropriate fix for the US's healthcare issues.

Unless healthcare businesses decide to improve patient care with AI instead of increasing patients per day, I think it's going to make things even worse.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#155

Post by jonfw »

andai wrote: post #147
dragonwriter wrote: post #126

Earlier quoted context omitted.

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans No, I don’t see that we must. > if we already have this assumption for software engineers No, this doesn’t follow, and even if it did, while I am aware that the CEOs…

Self-improving system given enough time to self-improve doesn't beat non-self-improving system?

Humans can certainly be self improving, both on an individual basis and in aggregate.

In humans, it seems that improvement in a new domain seems to follow a logarithmic scale.

Why wouldn’t this be the same for an AI?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#156

Post by emp17344 »

andai wrote: post #147
dragonwriter wrote: post #126

Earlier quoted context omitted.

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans No, I don’t see that we must. > if we already have this assumption for software engineers No, this doesn’t follow, and even if it did, while I am aware that the CEOs…

Self-improving system given enough time to self-improve doesn't beat non-self-improving system?

Currently that self-improving system isn’t so self-improving that it’s become better at any particular job than human beings, so I think the skepticism is warranted.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#157

Post by idiotsecant »

andai wrote: post #147
dragonwriter wrote: post #126

Earlier quoted context omitted.

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans No, I don’t see that we must. > if we already have this assumption for software engineers No, this doesn’t follow, and even if it did, while I am aware that the CEOs…

Self-improving system given enough time to self-improve doesn't beat non-self-improving system?

Please show me this self improving AI.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#158

Post by idiotsecant »

oofbey wrote: post #142
dragonwriter wrote: post #126

Earlier quoted context omitted.

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans No, I don’t see that we must. > if we already have this assumption for software engineers No, this doesn’t follow, and even if it did, while I am aware that the CEOs…

You’re holding on to the intuition (hope) that we are smarter than the LLMs in some hard to define way. Maybe. But it’s getting harder and harder to define a task that humans beat LLMs on. On pretty much any easily quantifiable test of knowledge or reasoning, the machines win. I agree experienced humans are still better on “judgement” tasks in their field. But the judgement tasks are kinda necessarily ones where ther…

There are almost no real world tasks that LLMs outperform humans on, operating by themselves. Pair them with a human for adaptability, judgement, and real world context and let the human drive, sure. Just let it loose on its own? You get an ocean of slop that doesn't do even close to what it's supposed to.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#159

Post by pianopatrick »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

Last time I went to the ER the doctor used a scope to look down my throat and check everything seemed fine. I don't think pure AI like ChatGPT will be able to do that any time soon. Maybe a medical robot with AI will one day, but that seems at least a few years off.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#160

Post by seanmcdirmid »

adamtaylor_13 wrote: post #152

Despite what I suspect the general consensus on HN may be, this does not surprise me at all. My wife was recently diagnosed with Mast Cell Activation Syndrome (MCAS) after a pretty scary series of ER visits. It's a very strange and stubborn autoimmune disease that manifests with a number of symptoms that, taken individually, could indicate damn near anything. You could almost feel the doctors rolling their eyes as sh…

The quality of doctors is really uneven, and the amount of things they can and have to pattern match on grows each year. I definitely hope they at least adopt AI tooling to ease their pattern matching burden. There is no reason AI needs to replace doctors, I think as it is in SWE doctors are still needed to guide and check the AI in its search for solutions.

Of course, there are plenty of places on earth that are extremely under doctored, and AI will definitely be better than nothing in poor regions of Africa if all it needs is a network connection and someone to donate the tokens.

Post reply on HN