Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

theguardian.com

Post reply on HN
Print view View original

121–130 of 500 posts

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#121

Post by ipaddr »

gherkinnn wrote: post #35
pixel_popping wrote: post #25

Earlier quoted context omitted.

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

Doctors are not necessarily great at talking to patients and patients are unhappy with the information Doctors provide. This moat has dried up.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#122

Post by Applejinx »

ianbutler wrote: post #59
gherkinnn wrote: post #35

Earlier quoted context omitted.

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

So much of what I know from women in my life is that the human element of medicine is almost a strict negative for them. As a guy it hasn't been much better, but at least doctors listen to me when I say something.

At which point I'd ask: how much of that is baked into the AI now?

It doesn't have opinions, research, direction of its own. Is this a path of codifying the worst elements of human society as we've known it, permanently?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#123

Post by fullstop »

ianbutler wrote: post #59
gherkinnn wrote: post #35

Earlier quoted context omitted.

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

So much of what I know from women in my life is that the human element of medicine is almost a strict negative for them. As a guy it hasn't been much better, but at least doctors listen to me when I say something.

Yes, yes, but when was your last period?

This even translates to the pediatric space. I took all of my kids to the pediatrician because either they don't make comments to me like they do to my wife, or I don't take shit from them. I'm not sure which. Here's an example:

My wife and daughter were there and the doctor asked what kind of milk my daughter was drinking. She said "whole milk" and the doctor made a comment along the lines of "Wow, mom, you really need to switch to 2%". To understand this, though, you need to understand that my daughter was _small_. Like they had to staple a 2nd sheet of paper to the weight chart because she was below the available graph space. It wasn't from lack of food or anything like that, she's just small and didn't have much of an appetite.

So I became the one to take the kids there. Instead of chastising me, they literally prescribed cheeseburgers and fettuccine alfredo.

My daughter is in her 20s now and is still small -- it's just the way she is. When she goes to see her primary, do you know what their first question is? "When was your last period."

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#124

Post by csomar »

gherkinnn wrote: post #35
pixel_popping wrote: post #25

Earlier quoted context omitted.

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

Doctors talk to patients?

I know. I know. Part of it is that talking to patients on average is useless but still this can’t be really used for an argument against AI.

Still doctors can have a more broad picture of the situation since they can look at the patient as a whole; something the LLM can’t really synthesize in its context.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#125

Post by prmph »

ddosmax556 wrote: post #117
gherkinnn wrote: post #35

Earlier quoted context omitted.

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

I think there's a real space there, and a lot of what e.g. nurses and doctors do is talking to humans, and that won't go away. But two facts are also true: a) diagnosis itself can be automated. A lot of what goes on between you having an achy belly and you getting diagnosed with x y or z is happening outside of a direct interaction with you - all of that can be augmented with AI. And b), the human interaction part is…

> Diagnosis becomes cheaper & easier -> more time to actually talk to patients

Unfortunately is this not likely to happen. More like:

Diagnosis becomes cheaper & easier -> more patients a doctor is expected to see in the same period of time as before

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#126

Post by dragonwriter »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

> After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans

No, I don’t see that we must.

> if we already have this assumption for software engineers

No, this doesn’t follow, and even if it did, while I am aware that the CEOs of firms who have an extraordinarily large vested personal and corporate financial interest in this being perceived to be the case have expressed this re: software engineers, I don’t think it is warranted there, either.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#127

Post by phoronixrly »

ipaddr wrote: post #121
gherkinnn wrote: post #35

Earlier quoted context omitted.

To answer your question: talking to a human. Medicine is so much more than "knowledge, experience, and pattern matching", as any patient ever can attest to. Why is it so hard for some people to understand that humans need other humans and human problems can't be solved with technology?

Doctors are not necessarily great at talking to patients and patients are unhappy with the information Doctors provide. This moat has dried up.

If you prefer an LLM to a human doctor, you deserve an LLM instead of a human doctor, and I wish you get it.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#128

Post by Applejinx »

tedggh wrote: post #68

Believable and not shocking. LLMs literally may have saved my sons and potentially her mother too by allowing us to fact check a lot of non sense data and scare tactics by a group of at least 5 different doctors ambushing us to make a life changing decision in minutes. The problem is doctors, at least in the US, prioritize liability exposure over patients long term outcomes. Let’s say you need an intervention where t…

Is the group of at least 5 different doctors ambushing you, in the room with us right now? Was it 5, or more like 15, or 50? Would it have been more or less frightening if it was a group of the same doctor, but like 40 of him?

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#129

Post by lokar »

gpm wrote: post #85
dyauspitr wrote: post #80

Earlier quoted context omitted.

I think the bigger takeaway here is that 50% of the time doctors will miss what you have.

That's not a takeaway here at all. It's 50% of the time ER doctors working solely from notes , something they never do, in a situation they know is only for a study, will miss what you have. In real clinical situations the doctors see, hear, smell, and interact with the patients.

Also, it just says they did not make the "correct" call, but that could mean they ordered an extra test, or took a more conservative route for treatment.

Re: OpenAI’s o1 correctly diagnosed 67% of ER patients vs. 50-55% by triage doctors

#130

Post by godelski »

pixel_popping wrote: post #25
gpm wrote: post #11

I'd be very very hesitant to trust studies like this. It's very easy to mess up these benchmarks. See for example this recent paper where AI managed to beat radiologists on interpreting x-rays... when the AI didn't even have access to the x-rays: https://arxiv.org/pdf/2603.21687 (on a pre existing "large scale visual question answering benchmark for generalist chest x-ray understanding" that wasn't intentionally mess…

I agree with you on this specific study, however, I can't really wrap my head about the fact that doctors will be better than AI models on the long-run. After all, medicine is all about knowledge, experience and intelligence (maybe "pattern recognition"), all those, we must assume that the best AI models (especially ones focusing solely in the medical field) would largely beat large majority of humans (aka doctors),…

  > After all, medicine is all about knowledge, experience and intelligence
So is... everything?

LLMs are really really good at knowledge.

But they are really really bad at intelligence [0]

They have no such thing as experience.

Do not fool yourself, intelligence and knowledge are not the same thing. It is extremely easy to conflate the two and we're extremely biased to because the two typically strongly correlate. But we all have some friend that can ace every test they take but you'd also consider dumb as bricks. You'd be amazed at what we can do with just knowledge. Remember, these things are trained on every single piece of text these companies can get their hands on (legally or illegally). We're even talking about random hyper niche subreddits. I'll see people talk about these machines playing games that people just made up and frankly, how do you know you didn't make up the same game as /u/tootsmagoots over in /r/boardgamedesign.

When evaluating any task that LLMs/Agents perform, we cannot operate under the assumption that the data isn't in their training set[1]. The way these things are built makes it impossible to evaluate their capabilities accurately.

[0] before someone responds "there's no definition of intelligence", don't be stupid. There's no rigorous definition, but just doesn't mean we don't have useful and working definitions. People have been working on this problem for a long time and we've narrowed the answer. Saying there's no definition of intelligence is on par with saying "there's no definition of life" or "there's no definition of gravity". Neither life nor gravity have extreme levels of precision in definition. FFS we don't even know if the gravaton is real or not.

[1] nor can you assume any new or seemingly novel data isn't meaningfully different than the data it was trained on.

Post reply on HN