Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models that predict disease are not as accurate as reports might suggest

scientificamerican.com

Post reply on HN
Print view View original

131–140 of 162 posts

Re: AI models that predict disease are not as accurate as reports might suggest

#131

Post by hef19898 »

ggm wrote: post #49
lostlogin wrote: post #32

Earlier quoted context omitted.

I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing. The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count dou…

My partner had a clinician review her paperwork and say "why are you here" explaining the enhanced imaging was leading to tentative concerns being raised about structural change so small it was below the threshold for safe surgical treatment. Moral of the story: the imaging has got so good that diagnostics is now on the fringe of over diagnosing and the stats need to catch up

One of the first things AI will be really good at will be image post processing. Even then I, in case of medical diagnosis, I'd prefer to have an actual person compare the "RAW" image to whatever the AI came up with. Simply because post processing can create artifacts that can throw you of quite a bit.

Regarding tue quality of imaging: I tend to agree, and the better imaging gets the more we will have to relly on humans to judge whether or not treatment is necessary or recommended. That judgement alone is, IMHO, in the same league as full self driving and requires general AI.

Re: AI models that predict disease are not as accurate as reports might suggest

#132

Post by srvmshr »

hef19898 wrote: post #125
srvmshr wrote: post #52

Earlier quoted context omitted.

> But I'm very curious how good one has to be in order to be better than a below-average doctor, or a 50th-percentile doctor, or a 75th. In dermatology, on which I was working, models were better (at detecting skin cancers) than 52% of the GPs, going by just images. In a famous Nature paper by Esteva et al., the TPR was at 74% for detecting Melanomas. There is a catch which probably got underreported (The skin cancer…

Isn't it a tad unfair to compare a ML model for dermatology, only working with pictures, against general practitioners? IMHO comparing said model against dermatologists would be a better approach. And just working from images is not necessarily a dermatological model, buy rather an image analysis model.

> We test its performance against 21 board-certified dermatologists

From Andre Esteva's paper:

https://www.nature.com/articles/nature21056

Re: AI models that predict disease are not as accurate as reports might suggest

#133

Post by gjulianm »

dekhn wrote: post #38
srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

Modern medicine already incorporates wide ranges of data. Doctors use flowcharts, scales, point systems, etc, to diagnose certain conditions because those tools have been developed by studying and considering a lot of cases.

However, there's a lot that isn't covered with data. The "middle of the scale", the "almost but not quite there", the "this is weird"... Doctors are good at that, through experience, and those are the difficult cases. Those are the ones where ML will not only likely fail, but won't even explain why it fails. We're talking about human lives here. If anything, I think software engineers massively overestimate the performance of ML and underestimate doctors.

Re: AI models that predict disease are not as accurate as reports might suggest

#134

Post by hef19898 »

kimi wrote: post #122

I feel it would be safe to say that "AI models that x are not as accurate as reports might suggest" for the current hype values of x.

AI seems to be a hyped solution in search for a problem. ML, sure, for some use cases. Regardless of where I encountered ML / AI / "expert systems in my life so far they have been sub-par to actual biological intelligence. In all cases they were really hyped.

Re: AI models that predict disease are not as accurate as reports might suggest

#135

Post by amelius »

bjt2n3904 wrote: post #66

This is what freaks me out about AI. People will use it for years in various fields, and one by one, after a decade or so of use, they'll come to find it was complete garbage information, and they were just putting their trust in a magic 8 ball. But the damage is already done.

Same with self-driving cars. State of the art AI-based classification has an accuracy of 90%. Even if we can get it to 99%, that's still 1% error. Now imagine a car taking hundreds of decisions in a single ride.

Re: AI models that predict disease are not as accurate as reports might suggest

#136

Post by ShamelessC »

baxtr wrote: post #36
NovemberWhiskey wrote: post #18

Earlier quoted context omitted.

Sorry for asking, but how is it not ?

Do you agree that it’s ok to pose a question whenever you don’t understand?

I’m not sure where you got this form of communication where you respond to everything with a question, and I assume you mean well, but it comes across as patronizing and de-humanizing to try to follow these “rules to winning arguments passively”, or whatever it is.

Indeed, the confusion here is (I think) because your first comment

> Sorry for asking, but how is this relevant to the article?

Sounds accusatory.

Please don’t respond to this with a question.

Re: AI models that predict disease are not as accurate as reports might suggest

#137

Post by dan_mctree »

dm319 wrote: post #7

As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…

But doctors tend to be pretty good at that part of the job you're describing. It's the Dr House part that often seems lacking, difficult patients often spend years going to different doctors only to keep hearing useless suggestions. It'd be amazing if we could have a solution for those people and AI might be able to help

Re: AI models that predict disease are not as accurate as reports might suggest

#138

Post by Gatsky »

srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

For ML to really make a dent in medicine, the whole system needs to be altered, in a similar vein to how building roads tailored to self driving cars would make them much more successful. Most medical diagnoses are only made when severe physiological derangement has already occurred. If we had access to longitudinal streams of data, then ML would be essential to detecting anomalies which point to early evidence of disease. For example, streaming in regular noninvasive measurements on breath and urine; wearable readouts on heart rate, oxygen saturation, blood pressure, temperature, movement; neurocognitive streams based on analysis of email, video calls, text messages etc. But this is a fundamental change on many levels, with many barriers. It will also probably fail to improve the health of people who need it the most. I’m not sure it’s even that appealing as an alternative to the current meatspace system.

Re: AI models that predict disease are not as accurate as reports might suggest

#139

Post by rickysahu »

we are working on some transformer based models trained on millions of patients across a broad set of features. Well launch our first model into a private alpha in a few weeks. Basically it uses a seq2seq transformer XL and fhir data. https://labs.1up.health/ Were hiring. U can email me directly at ricky@ We don't do any of the subset optimization or training for specific diseases or conditions. It's all or nothing like the way most llms work

Re: AI models that predict disease are not as accurate as reports might suggest

#140

Post by davikr »

gjulianm wrote: post #133
dekhn wrote: post #38

Earlier quoted context omitted.

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

Modern medicine already incorporates wide ranges of data. Doctors use flowcharts, scales, point systems, etc, to diagnose certain conditions because those tools have been developed by studying and considering a lot of cases. However, there's a lot that isn't covered with data. The "middle of the scale", the "almost but not quite there", the "this is weird"... Doctors are good at that, through experience, and those ar…

Yes, notwithstanding those factors you described, it is not uncommon for tests to reveal false-negative or false-positive results due to their intrinsic specificity and sensitivity. A normal value is not always indicative of health, either.
Post reply on HN