I’m shocked. SHOCKED.
AI models that predict disease are not as accurate as reports might suggest
31–40 of 162 posts
Re: AI models that predict disease are not as accurate as reports might suggest
#32As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…
I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing.
The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count doubles in x and y dimensions).
These are 2 distinct processes which we can turn on or off and have some parameters which with we can alter the process.
The result is amazing and image quality has gone up a lot.
We haven’t got a full grasp yet and have a few theories. The vendors are also still getting to grips.
We think the training data set turns out to have some weird influences on requires acquisition parameters. For example, parallel imaging factor 4 works well, 3 and 2 less so, which is not intuitive. More acceleration being better for image quality is not how MRI used to work (except in a few edge cases).
Bandwidth, averages, square pixel, turbo factor and appropriate TE matter a bit more than they did pre-AI.
Images are now acquired faster, look better and sequence selection can be better tailored to the patient as we have less of a time pressure.
I’d put our images up against almost anything I’ve seen before as examples of good work. We are seeing anatomy and pathology that we didn’t previously appreciate. Sceptics ask if the things we see are really there, but after some time with the images the concern goes away and the pre-AI images just look broken.
In the below link, ignore Gain (it isn’t that great), Boost and Sharp are the vendor names for the good stuff. The brochure undersells it.
https://www.siemens-healthineers.com/magnetic-resonance-imag...
Re: AI models that predict disease are not as accurate as reports might suggest
#33My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. An attempt to dispense with mathematicians to develop models. What amount of energy is necessary for an event to be reflected in a statistic? You have a box of 2x2 meters with ba…
[deleted]
Re: AI models that predict disease are not as accurate as reports might suggest
#34I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company.
My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X, physicians factor in blood work, family history, imaging, racial genealogy, general symptoms (like hoarseness, gait, sweating etc), even texture & palpitations of affected regions sometimes before narrowing down on a set of assessments & making diagnostic decisions.
If we just add in more dimensions of data to model, it just makes the search space sparser, not easier. Throwing in more data will likely just fit more common patterns & classes well, whereas a large number of symptoms may be treated as outliers and mispredicted.
We humans are incredibly good at elimination of factors & differential diagnosis. The findings don't surprise me. There is much more work needing to be covered. For straightforward, and conditions with limited, clear cut symptoms they are showing promising advancements, but it cannot be trusted to wide arrays of diagnosis - especially when models don't know what 'they do not know'.
Re: AI models that predict disease are not as accurate as reports might suggest
#35Earlier quoted context omitted.
Sorry for asking, but how is this relevant to the article?
Fair question. The model we comment on both suffer from the problem described in the article but also a more severe problem: The developers sampled obvious cases og hypotension and nonhypotension, and trained the model to distinguish those. And also validated it on data that was similarly dichotomous. In reality the outcome is often between these two scenarios. But worse, they also introduce a more severe problem whe…
Thanks for explaining!
Re: AI models that predict disease are not as accurate as reports might suggest
#36Re: AI models that predict disease are not as accurate as reports might suggest
#37'Brunelleschi had just the solution. To get around the issue, the contest contender proposed building two domes instead of one — one nested inside the other. "The inner dome was built with four horizontal stone and chain hoops which reinforced the octagonal dome and resisted the outward spreading force that is common to domes, eliminating the need for buttresses," Wildman says. "A fifth chain made of wood was utilized as well. This technique had never been utilized in dome construction before and to this day is still regarded as a remarkable engineering achievement.'
Brunelleschi was not an engineer he was a goldsmith. AI will advance in the same way architecture did during the Renaissance. By those with the winning ideas not with the right credentials.
https://science.howstuffworks.com/engineering/architecture/b...
Re: AI models that predict disease are not as accurate as reports might suggest
#38I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…
are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better?
I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.
Re: AI models that predict disease are not as accurate as reports might suggest
#39Earlier quoted context omitted.
From the article: > Here’s why: As researchers feed data into AI models, the models are expected to become more accurate, or at least not get worse. However, our work and the work of others has identified the opposite, where the reported accuracy in published models decreases with increasing data set size.
That's not a contradiction per se. It's easier to get spurriously high test scores with smaller datasets. It does not clearly demonstrate that the models are actually getting worse.
But if diagnosis are multimodal and rely upon large, multidimensional analysis of symptoms/bloodwork/past medical history, wouldn't adding more dimensions just increase dimensional sparsity and decrease the useful amount of conclusions you are able to draw from your variables?
It's been a long time since I remember learning about the curse of dimensionality but if you increase the amount of datapoints you collect by half you would have to quadruple the amount of samples you have to retrieve any meaningful benefit, no?
Re: AI models that predict disease are not as accurate as reports might suggest
#40My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. An attempt to dispense with mathematicians to develop models. What amount of energy is necessary for an event to be reflected in a statistic? You have a box of 2x2 meters with ba…
Statistics is not science- it's an application of probability theory and some other forms of math to hypothesis selection (among other things).
It's scientific. We only use stats because that's the best method for dealing with imprecise and noisy data.
Statistical thermodynamics contains all the necessary tools you need to answer your balls in a box question.