Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models that predict disease are not as accurate as reports might suggest

scientificamerican.com

Post reply on HN
Print view View original

91–100 of 162 posts

Re: AI models that predict disease are not as accurate as reports might suggest

#91

Post by mjthrowaway1 »

johannes_ne wrote: post #23
baxtr wrote: post #16

Earlier quoted context omitted.

Sorry for asking, but how is this relevant to the article?

Fair question. The model we comment on both suffer from the problem described in the article but also a more severe problem: The developers sampled obvious cases og hypotension and nonhypotension, and trained the model to distinguish those. And also validated it on data that was similarly dichotomous. In reality the outcome is often between these two scenarios. But worse, they also introduce a more severe problem whe…

I quit research forever after I was ignored pointing out a similar problem in our predictive model.

Re: AI models that predict disease are not as accurate as reports might suggest

#92

Post by p1necone »

lostlogin wrote: post #32
dm319 wrote: post #7

As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…

I work in radiology with MRI as a tech. We use AI slightly differently to the examples here, but it’s changing a lot of what we do. It’s more about enhancing images than directly about diagnosing. The image is denoised ‘intelligently’ in k-space and then the resolution is doubled via another AI process in the image domain (or maybe the resolution is quadrupled, as it depends on how you measure it. Our pixel count dou…

AI denoising may be making information that actually is in the image easier to spot, but AI upscaling is just inventing detail that doesn't actually exist in the source image, which seems rather dangerous for this use case.

Re: AI models that predict disease are not as accurate as reports might suggest

#93

Post by ThePhantom »

srvmshr wrote: post #45
dekhn wrote: post #38

Earlier quoted context omitted.

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

> Are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? Personal story: I was diagnosed with a rare genetic disease in 2019. If I ran the symptoms through a ML gauntlet, I would be sure they would cancel each other out or make little sense. Chest CT (clean), fever (high), TB test (negative), laten…

What is the name of this disease?

Re: AI models that predict disease are not as accurate as reports might suggest

#94

Post by buscoquadnary »

srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

It's interesting you say this. I read a book several years ago, I don't remember what it was. But it talked about how there used to be a lot of questions that were used to determine if someone was having a heart attack.

They then did a lot of stats and number crunching and determined that with just 2 questions they could accurately predict whether or not someone was having a heart attack 95% of the time which was a considerable improvement.

I wonder how many problems are we making worse by throwing more data at rather than making them better?

Re: AI models that predict disease are not as accurate as reports might suggest

#95

Post by naijaboiler »

dekhn wrote: post #38
srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

Not this ignorant comment again. AI will replace software engineers long before it replaces doctors. There is an arrogant ignorance of what doctors do that always shows up in comments when topics like this pop up.

And yes I'm a physician and MLE. So i understand both worlds clearly

Re: AI models that predict disease are not as accurate as reports might suggest

#96

Post by buscoquadnary »

nostromo wrote: post #82
srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

I’m confused by your comment because these are exactly the type of problems that humans generally really do a poor job classifying.

It could be that after we train a biological nural net for decades it can get pretty good at intuiting things even if it can't explain how.

The numeral net in question is the Drs. Brain.

Re: AI models that predict disease are not as accurate as reports might suggest

#97

Post by buscoquadnary »

esel2k wrote: post #63
dm319 wrote: post #7

As someone who works in healthcare, so much of what I read about AI makes me think that the people who are enthusiastic about healthcare AI don't have much experience doing it. The scenarios rarely seem to fit with what I'm actually practicing. Most of medicine is boring, it is largely routine, and if we don't know what's going on, it's because we're not the right person to be managing the patient. Most of my time is…

So much this. I just interviewed about 10 doctors in the space of neurology and radiology to start some new projects. The truth is most of the headaches are from insurance coverage check or for radiologist for filling out correct reports. The fancy AI stuff is with maybe a few exceptions due to the great advancement imaging still far away from validation and I didn’t even start about it’s usage and gotomarket. Most o…

> and problems like access to medical history is way more basic but more prevalent.

This here is a problem I wish we could solve, get a central data broker that could hold aggregate medical history and just dole it out to companies. My wife has to get Medical supplies on a regular basis, the company we got them from got bought out by a PEC so we wanted to try switching companies. Months later we still can't get all the Drs insurance and supplier lined up.

The problem is of course as a security guy I know a centralized clearinghouse would be a nightmare from every stand point. The real solution may be in the development of a good protocol, but good luck getting the Med industry that is still putzing around on ASM/400s to adopt something like that. Even with the force of law, i.e. HIPPA

Re: AI models that predict disease are not as accurate as reports might suggest

#98

Post by naijaboiler »

the_d3f4ult wrote: post #88
esel2k wrote: post #63

Earlier quoted context omitted.

So much this. I just interviewed about 10 doctors in the space of neurology and radiology to start some new projects. The truth is most of the headaches are from insurance coverage check or for radiologist for filling out correct reports. The fancy AI stuff is with maybe a few exceptions due to the great advancement imaging still far away from validation and I didn’t even start about it’s usage and gotomarket. Most o…

I'm a physician and former engineer and I agree with you. Even the giant tech companies that are in this space seem to have very little insight into how to utilize AI to improve healthcare. The mistake everyone seems to make is thinking that making a diagnosis is what makes medicine hard. It's not. e.g. An algorithm that can diagnose diabetic retinopathy from fundus photos 'as well as a specialist' makes for a good h…

I can't upvote this enough

Re: AI models that predict disease are not as accurate as reports might suggest

#99

Post by potatoman22 »

majormajor wrote: post #50
dekhn wrote: post #38

Earlier quoted context omitted.

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

My view on this is framed a bit differently but probably a similar ultimate perspective: I think it's probably going to be a long time before models only using quantifiable measurements can even meet the performance of top doctors. I can't recommend enough that someone experiencing issues doctor-shop if they haven't gotten a well-explained diagnosis from their current doctor. But I'm very curious how good one has to…

Statistical mortality models are already more accurate than physicians for an average patient.

Re: AI models that predict disease are not as accurate as reports might suggest

#100

Post by naijaboiler »

bitL wrote: post #53

OK, AI is bad but compare it to human doctors/radiologists that are often worse. I still remember stats from some X-ray detection where AI diagnosed with 40% accuracy and the best human doctors with 38% accuracy (and median human doctors with 32% accuracy). Now what are we supposed to do?

Almost any paper like this is worthless. It makes for good headlines. But it does nothing to further medical care
Post reply on HN