Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models that predict disease are not as accurate as reports might suggest

scientificamerican.com

Post reply on HN
Print view View original

61–70 of 162 posts

Re: AI models that predict disease are not as accurate as reports might suggest

#61

Post by srvmshr »

pixl97 wrote: post #59
srvmshr wrote: post #45

Earlier quoted context omitted.

> Are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? Personal story: I was diagnosed with a rare genetic disease in 2019. If I ran the symptoms through a ML gauntlet, I would be sure they would cancel each other out or make little sense. Chest CT (clean), fever (high), TB test (negative), laten…

How many specialists did you go to before it was identified? How many other people with the condition were misidentified? I only say this because of a family member with a rare genetic condition. For years they were told it was something else, or told 'it was in their head'. The family member started a journal of their medical conditions and experiences that was detailed then brought that to their PC which whom then…

> How many specialists did you go to before it was identified?

2 opthalmo, 1 internal medicine, 1 retina super-specialist & finally someone from USC Davey

> How many other people with the condition were misidentified?

Historical data: I don't know. It is fairly divided between two types, one being zoonotic & other to IL2 gene. I am told this distinction of pathways was identified in 2007.

> [..] you kept a detailed list (on paper or in your mind) of the aliments and presented them in a manner that helped with the final diagnosis.

I might have been a better informed patient but I went with a complaint of pink eye, flu & mild light sensitivity. Never imagined that visit would change my life forever. Thank you though, for expressing your concern & support

Re: AI models that predict disease are not as accurate as reports might suggest

#62

Post by dekhn »

dirheist wrote: post #41
planetsprite wrote: post #19

The solution to failures of AI in heathcare is transparency of data. OpenAI's models work because they have virtually unlimited data to train on. The scale of training data for doctor bots is one millionth the size. Different countries, organizations, universities need to be as open as possible sharing and collaborating, realizing improvements in medicine benefits all of humanity with almost no downsides.

There should be a standardization committee tasked with standardizing the collection of anonymized, semi-synthetic medical data from hospitals/hospital networks. It seems like so much research is just locked up in the IMS systems the hospitals use for their patients and that never see the light of day.

You cannot imagine just how deep the medical data rabbit hole goes.

Already plenty of institutions have semi-standardized their collect and do multi-hospital (typically research hospitals) aggregation. Whether this data is any good as training data for supervised or unsupervised algorithms is really questionable.

Re: AI models that predict disease are not as accurate as reports might suggest

#63

Post by esel2k »

dm319 wrote: post #7

As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…

So much this. I just interviewed about 10 doctors in the space of neurology and radiology to start some new projects. The truth is most of the headaches are from insurance coverage check or for radiologist for filling out correct reports. The fancy AI stuff is with maybe a few exceptions due to the great advancement imaging still far away from validation and I didn’t even start about it’s usage and gotomarket.

Most of the cases the doctors sees are boring / regular cases - and problems like access to medical history is way more basic but more prevalent.

Re: AI models that predict disease are not as accurate as reports might suggest

#64

Post by srvmshr »

lostlogin wrote: post #32
dm319 wrote: post #7

As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…

I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing. The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count dou…

I did my Masters in NMR. Can confirm a lot of ML based plug-and-play solutions are helping denoising k-space.

Trivia: I am also one of the pulse sequence developers affiliated to Siemens LiverLab package on Syngo platform :) [Specifically the multiecho Dixon fat-water sequence]. SNR improvement was a big headache for rapid Dixon echos.

Re: AI models that predict disease are not as accurate as reports might suggest

#65

Post by esel2k »

lostlogin wrote: post #46
ckemere wrote: post #27

Earlier quoted context omitted.

There's also the timeline that: "Radiology will be automatized in 5 years" (10 years ago) "Radiology will be automatized in 5 years" (5 years ago) "Radiology will be automatized in 5 years" (last year) or "Full self driving will arrive within 5 years" (5 years ago) "Full self driving is still a ways off" (last year) Assuming you're referring to generative models, I don't think that anyone (knowledgable) thinks that g…

Having seen some of the automation available in radiology, I’m a bit baffled as to why I still have a job as an MRI tech. 5 years ago I watched automated cardiac MRI, and it worked well. I was told about a site that were having good results with fetal cardiac MRI via a related bit of software. These scans are hard to do, and the machines did well. In some cases they got confused and did a good functional analysis but…

But even if all these analysis would be dine automatically, I guess you won’t be out of a job soon (good news I guess). But just different: I did in the automation of the diagnostic lab and what happened is that from a detective style job, today it is more about running a factory. 24h running a business, turn around times and have less and less qualified personnel to fill the machines…

Re: AI models that predict disease are not as accurate as reports might suggest

#66

Post by bjt2n3904 »

This is what freaks me out about AI.

People will use it for years in various fields, and one by one, after a decade or so of use, they'll come to find it was complete garbage information, and they were just putting their trust in a magic 8 ball.

But the damage is already done.

Re: AI models that predict disease are not as accurate as reports might suggest

#68

Post by lostlogin »

srvmshr wrote: post #64
lostlogin wrote: post #32

Earlier quoted context omitted.

I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing. The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count dou…

I did my Masters in NMR. Can confirm a lot of ML based plug-and-play solutions are helping denoising k-space. Trivia: I am also one of the pulse sequence developers affiliated to Siemens LiverLab package on Syngo platform :) [Specifically the multiecho Dixon fat-water sequence]. SNR improvement was a big headache for rapid Dixon echos.

Ha, small world. Thanks for your work, I used to use this daily until a year ago, now my usage is less frequent.

I guess Dixons are still a headache with their new k-space stuff as Boost (the denoising) isn’t compatible with it yet. Gain is but looks distinctly lame when you compare it Boost.

We are yet to see the tech applied to breath hold sequences (haste, vibe etc), Dixon, 3D, gradient sequences and probably others.

I’m looking forward to seeing it on haste and 3D T2s (space) in particular. MRI looks very different today compared to how it looked just 6 months ago.

I’d compare it to the change we saw going from 1.5T to 3T, just accelerated in how quickly progress is being made.

Re: AI models that predict disease are not as accurate as reports might suggest

#69

Post by cm2187 »

So a model calibrated on a backtest says nothing about its predictive capacity. Who would have thought? Well, at least I think anyone who worked even a little bit in quantitative finance. The only way to validate a model is to make predictions and test if those predictions actually happen in a repeatable way, which in certain circles is refered to as "experiment".

That's why I distrust any model built purely on backtested data unless they can be shown to predict something else than history. And AI is not the only area that blindly trusts those kind of models.

Re: AI models that predict disease are not as accurate as reports might suggest

#70

Post by dmurray »

pcurve wrote: post #57
bitL wrote: post #53

OK, AI is bad but compare it to human doctors/radiologists that are often worse. I still remember stats from some X-ray detection where AI diagnosed with 40% accuracy and the best human doctors with 38% accuracy (and median human doctors with 32% accuracy). Now what are we supposed to do?

Can you cite the source? Is it not possible to improve the 40% rate by AI? Obviously someone eventually figured out the 100%

They might have "figured it out" by cutting the patient open.
Post reply on HN