Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models that predict disease are not as accurate as reports might suggest

scientificamerican.com

Post reply on HN
Print view View original

151–160 of 162 posts

Re: AI models that predict disease are not as accurate as reports might suggest

#151

Post by rowanG077 »

srvmshr wrote: post #45
dekhn wrote: post #38

Earlier quoted context omitted.

are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? I suspect we massively underestimate the amount of misdiagnosis due to incorrect analysis of data using fairly naive medical mental models of disease.

> Are you really sure the doctors are doing a better job when they go through the motions of incorporating a wide range of data? Or do we just convince ourselves they're better? Personal story: I was diagnosed with a rare genetic disease in 2019. If I ran the symptoms through a ML gauntlet, I would be sure they would cancel each other out or make little sense. Chest CT (clean), fever (high), TB test (negative), laten…

Maybe I'm reading it wrong but doesn't your story confirm that doctors aren't good at diagnoses? It took the cream of the crop top tier specialist to correctly diagnose your condition.

Re: AI models that predict disease are not as accurate as reports might suggest

#152

Post by isitmadeofglass »

baxtr wrote: post #141
ShamelessC wrote: post #136

Earlier quoted context omitted.

I’m not sure where you got this form of communication where you respond to everything with a question, and I assume you mean well, but it comes across as patronizing and de-humanizing to try to follow these “rules to winning arguments passively”, or whatever it is. Indeed, the confusion here is (I think) because your first comment > Sorry for asking, but how is this relevant to the article? Sounds accusatory. Please…

The basic idea of that kind of question is to find the minimal place of agreement. And then understand where one deviates. Going back the path of arguments to common ground if you will. It works quite well in my experience if you’re interested in genuine discussion. PS: how something “sounds” is really difficult to say in a written medium. It might say more about the reader than the writer.

> PS: how something “sounds” is really difficult to say in a written medium. It might say more about the reader than the writer.

No, it’s not difficult. And not it’s not the reader. When multiple readers all agree about the same interpretation of the writer. It might have been unintentional on the part of the writer, but that doesn’t make it “difficult” Or the readers fault.

Re: AI models that predict disease are not as accurate as reports might suggest

#153

Post by hef19898 »

lostlogin wrote: post #146
hef19898 wrote: post #131

Earlier quoted context omitted.

One of the first things AI will be really good at will be image post processing. Even then I, in case of medical diagnosis, I'd prefer to have an actual person compare the "RAW" image to whatever the AI came up with. Simply because post processing can create artifacts that can throw you of quite a bit. Regarding tue quality of imaging: I tend to agree, and the better imaging gets the more we will have to relly on hum…

The raw image is not good and isn’t usable. AI denoises and this is what makes it usable. Then it doubles the resolution. There is no point in reviewing the raw image as it doesn’t add anything. A study is generally 300-1000 images. If you’re going to review the raw and therefore look at 600-2000 images you’ve just wasted everything AI gained and you might as well not use it. I acquire the images and I look at what I…

Thanks for the explanation! Shows that some knowledge in digital photography doesn't translate into other domains taking pictures.

Comments like yours are what I love about HN!

Re: AI models that predict disease are not as accurate as reports might suggest

#154

Post by Melting_Harps »

srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

> We humans are incredibly good at elimination of factors & differential diagnosis. The findings don't surprise me. There is much more work needing to be covered. For straightforward, and conditions with limited, clear cut symptoms they are showing promising advancements, but it cannot be trusted to wide arrays of diagnosis - especially when models don't know what 'they do not know'.

If it were presented this way, while accurate and honest, it would in no way get the media hype and thus funding from both state actors and private investors looking to get to be a part of the 'winner take all' model.

As a person studying AI and ML at the undergrad level, is there any advice you have in order to the pitfalls that this Industry has become?

Re: AI models that predict disease are not as accurate as reports might suggest

#155

Post by ShamelessC »

baxtr wrote: post #141
ShamelessC wrote: post #136

Earlier quoted context omitted.

I’m not sure where you got this form of communication where you respond to everything with a question, and I assume you mean well, but it comes across as patronizing and de-humanizing to try to follow these “rules to winning arguments passively”, or whatever it is. Indeed, the confusion here is (I think) because your first comment > Sorry for asking, but how is this relevant to the article? Sounds accusatory. Please…

The basic idea of that kind of question is to find the minimal place of agreement. And then understand where one deviates. Going back the path of arguments to common ground if you will. It works quite well in my experience if you’re interested in genuine discussion. PS: how something “sounds” is really difficult to say in a written medium. It might say more about the reader than the writer.

I think what youre trying to encourage is open ended discussion? It's my opinion that this only tends to work IRL or in online mediums with more moderation e.g. wikipedia, stackoverflow.

Random open ended discussion can be good, but I bet it's wise to assume tht most random musings arent really as interesting as you might think.

In any case thanks for clarifying.

Re: AI models that predict disease are not as accurate as reports might suggest

#157

Post by pas »

johannes_ne wrote: post #144
pas wrote: post #142

Earlier quoted context omitted.

Hm, reading the linked tweets the problem seems like a big screaming red target on the side of a white barn, not a feature engineering subtlety. It seems like the typical case of the drunk guy looking for his keys under the streetlight. (Having insufficient data, and comparing the model to an arbitrarily picked one that just happens to be even worse. And then everyone including the FDA patting them on the back.)

I'm glad that you seem to get the severity! I'm just hesitant to ascribe malice.

I think it is the general incompetence of the "academia + R&D biz + regulation pipeline". (In the land of the blind the one-eyed is king, etc.)

It's sort of inevitable in such a non-teleological process. As in each step in it serves its own purpose, and so the whole thing doesn't really serve the purpose that we like to assume for it - ie. give us great thoughtful inventions. That's why it took so long to stop the Theranos train, that's why it takes so fucking long to roll out polyvalent vaccines (ie. all-in-one vaccines), and so on. (I'm picking on medtech here but there are many others, the Boeing + FAA MCAS fuckup, the absolute limpdick paralysis of nuclear power - it needed a combination of half the world on fire + prelude-to-WWIII to get it moving again, and so on.)

Re: AI models that predict disease are not as accurate as reports might suggest

#158

Post by Dr3xus »

mpaepper wrote: post #148

I work in machine learning for digital Pathology and I think the big problem here is the divergence between publishing papers and real life helpful models. What we see is that in the literature you often see a model trained on data from a single lab and gets crazy good results. However, apply it to a different lab (not in the paper of course) and it sucks. So what we do in practice for our models is to train on many…

Do you have any specific example use cases where you're seeing success?

I'm asking this both with regards to ML/prediction related work and the "boring" work.

I'm working on a direct to consumer digital pathology service with a clinical background as opposed to data science/ML. Really curious as to what type of new services we could try invest in to improve what we offer.

Re: AI models that predict disease are not as accurate as reports might suggest

#159

Post by dirheist »

tappio wrote: post #48
dirheist wrote: post #39

Earlier quoted context omitted.

But if diagnosis are multimodal and rely upon large, multidimensional analysis of symptoms/bloodwork/past medical history, wouldn't adding more dimensions just increase dimensional sparsity and decrease the useful amount of conclusions you are able to draw from your variables? It's been a long time since I remember learning about the curse of dimensionality but if you increase the amount of datapoints you collect by…

You are right, but I feel you misunderstood op. I understood that op meant increase number of samples, not variables.

I think I did misunderstand you are right, definitely increasing the number of samples will increase the feasibility of the model I was incorrect

Re: AI models that predict disease are not as accurate as reports might suggest

#160

Post by roflyear »

johannes_ne wrote: post #144
pas wrote: post #142

Earlier quoted context omitted.

Hm, reading the linked tweets the problem seems like a big screaming red target on the side of a white barn, not a feature engineering subtlety. It seems like the typical case of the drunk guy looking for his keys under the streetlight. (Having insufficient data, and comparing the model to an arbitrarily picked one that just happens to be even worse. And then everyone including the FDA patting them on the back.)

I'm glad that you seem to get the severity! I'm just hesitant to ascribe malice.

One thing I've learned, is people are definitely morons, so you're probably making the correct decision here.
Post reply on HN