Earlier quoted context omitted.
Sorry for asking, but how is it not ?
Do you agree that it’s ok to pose a question whenever you don’t understand?
Ironically, that's exactly what NovemberWhiskey is doing here :)
AI models that predict disease are not as accurate as reports might suggest
51–60 of 162 posts
Re: AI models that predict disease are not as accurate as reports might suggest
#52Earlier quoted context omitted.
are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.
My view on this is framed a bit differently but probably a similar ultimate perspective: I think it's probably going to be a long time before models only using quantifiable measurements can even meet the performance of top doctors. I can't recommend enough that someone experiencing issues doctor-shop if they haven't gotten a well-explained diagnosis from their current doctor. But I'm very curious how good one has to…
> But I'm very curious how good one has to be in order to be better than a below-average doctor, or a 50th-percentile doctor, or a 75th.
In dermatology, on which I was working, models were better (at detecting skin cancers) than 52% of the GPs, going by just images. In a famous Nature paper by Esteva et al., the TPR was at 74% for detecting Melanomas. There is a catch which probably got underreported (The skin cancer positivity rate was strongly correlated to clinical markings in photos. Their models didn't do quite as well when 'clean', holdout data were used).
But the nature of information in all these models were skin deep (pun intended). They were designed with a calibrated objective in place unlike how we approach clinical diagnostics as open ended problems for the doctors.
Re: AI models that predict disease are not as accurate as reports might suggest
#53OK, AI is bad but compare it to human doctors/radiologists that are often worse. I still remember stats from some X-ray detection where AI diagnosed with 40% accuracy and the best human doctors with 38% accuracy (and median human doctors with 32% accuracy). Now what are we supposed to do?
Re: AI models that predict disease are not as accurate as reports might suggest
#54As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…
[deleted]
Re: AI models that predict disease are not as accurate as reports might suggest
#55Earlier quoted context omitted.
I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing. The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count dou…
My partner had a clinician review her paperwork and say "why are you here" explaining the enhanced imaging was leading to tentative concerns being raised about structural change so small it was below the threshold for safe surgical treatment. Moral of the story: the imaging has got so good that diagnostics is now on the fringe of over diagnosing and the stats need to catch up
This has been a thing for a long time, with MRI in particular.
It gets quite philosophical. To diagnose something you need some pattern on the images. As resolution and tissue contrast improves you see more things, and the radiologist gets to decide if the appearance is something.
When a clinician says there is a problem in some area of anatomy and there is something on the scan, the radiologist has to make a call.
The great thing about being a tech is that making the call isn’t my job. I have noticed that keeping the field of view small tends to make me more friends.
A half imaged liver haemangioma, a thyroid nodule or a white matter brain lesion as an incidental finding are a daily occurrence at least.
Re: AI models that predict disease are not as accurate as reports might suggest
#56My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. An attempt to dispense with mathematicians to develop models. What amount of energy is necessary for an event to be reflected in a statistic? You have a box of 2x2 meters with ba…
It's not that AI has been conflated with machine learning—those are words that are supposed to refer to the same thing. The confusion is conflating either with slapdash applied statistics.
Re: AI models that predict disease are not as accurate as reports might suggest
#57OK, AI is bad but compare it to human doctors/radiologists that are often worse. I still remember stats from some X-ray detection where AI diagnosed with 40% accuracy and the best human doctors with 38% accuracy (and median human doctors with 32% accuracy). Now what are we supposed to do?
Can you cite the source? Is it not possible to improve the 40% rate by AI? Obviously someone eventually figured out the 100%
Re: AI models that predict disease are not as accurate as reports might suggest
#58Technical (Honest) Solution: two holdouts 1. Involved in the build process 2. Never touched until paper metrics are being written, only run once Realistically, unlikely to occur however due to the incentives causing publication bias.
Third (better) option: have a regulating body have a separate, undisclosed test set. If you can't beat it, you can't deploy your model. If you can beat it, you still need to have your models peer reviewed and scrutinized
This sounds simple yet I expect data governance will be the bottleneck.
Re: AI models that predict disease are not as accurate as reports might suggest
#59Earlier quoted context omitted.
are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.
> Are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? Personal story: I was diagnosed with a rare genetic disease in 2019. If I ran the symptoms through a ML gauntlet, I would be sure they would cancel each other out or make little sense. Chest CT (clean), fever (high), TB test (negative), laten…
How many specialists did you go to before it was identified?
How many other people with the condition were misidentified?
I only say this because of a family member with a rare genetic condition. For years they were told it was something else, or told 'it was in their head'. The family member started a journal of their medical conditions and experiences that was detailed then brought that to their PC which whom then sent them to a specialist, this specialist wasn't sure and sent them to another specialist that had a 3 month wait. After 5+ years of living with increasing severity of the condition it was identified.
So, just saying, it's as much likely that the condition was identified because you kept a detailed list (on paper or in your mind) of the aliments and presented them in a manner that helped with the final diagnosis.
Re: AI models that predict disease are not as accurate as reports might suggest
#60It is still unclear to me exactly what data they were looking at/referring to in this article.
If you take into account bloodwork, family history, demographics, etc. then it seems like you are still only getting a few dozen data points. At this scale it seems like traditional statistics or human checks for abnormalities are going to be about as good.
Although I personally know very little (apologies for conjecturing) it does seem like there could be a lot of uses for AI for specific diagnosis. For example, when they take your blood pressure/heartbeat they only get data for one particular moment where you are sitting in a controlled environment. I would think if you had a year's worth of data (along with activity data from an apple watch) you might be able to diagnose/predict things that traditional doctors/human analysis could not.
I would also imagine anything that deals with image analyzing (like looking for tumors in scans) will be vastly better with computer AI systems than humans.