Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI models that predict disease are not as accurate as reports might suggest

scientificamerican.com

Post reply on HN
Print view View original

71–80 of 162 posts

Re: AI models that predict disease are not as accurate as reports might suggest

#71

Post by fallingknife »

srvmshr wrote: post #34

I worked in healthcare ML solutions, as part of my PhD & also as consultant to a telemedicine company. My experience in dealing with data (we had sufficient, and somewhat well labeled) & methods made me realize that a lot of the prediction human doctors make are multimodal - and that is something deep learning will struggle for the time being. For example, say in detection of a disease X , physicians factor in blood…

The system itself should be built around these capabilities, not the other way around. Instead of collecting data at regular intervals we wait until symptoms to go to the doctor. This is why the dataset is so sparse.

Re: AI models that predict disease are not as accurate as reports might suggest

#72

Post by cm2187 »

tomrod wrote: post #20

Technical (Honest) Solution: two holdouts 1. Involved in the build process 2. Never touched until paper metrics are being written, only run once Realistically, unlikely to occur however due to the incentives causing publication bias.

so the models that fail this one test never get published and the models that succeed get published. And all you have done is to publish a model that predicts that particular history, in other words data fitting.

Re: AI models that predict disease are not as accurate as reports might suggest

#73

Post by srvmshr »

lostlogin wrote: post #68
srvmshr wrote: post #64

Earlier quoted context omitted.

I did my Masters in NMR. Can confirm a lot of ML based plug-and-play solutions are helping denoising k-space. Trivia: I am also one of the pulse sequence developers affiliated to Siemens LiverLab package on Syngo platform :) [Specifically the multiecho Dixon fat-water sequence]. SNR improvement was a big headache for rapid Dixon echos.

Ha, small world. Thanks for your work, I used to use this daily until a year ago, now my usage is less frequent. I guess Dixons are still a headache with their new k-space stuff as Boost (the denoising) isn’t compatible with it yet. Gain is but looks distinctly lame when you compare it Boost. We are yet to see the tech applied to breath hold sequences (haste, vibe etc), Dixon, 3D, gradient sequences and probably othe…

I have long since left collaboration with team at Cary, NC. But all I can say there was a great deal of interest in 3D sequence improvement by interpolation with known k-space patterns like in the GRASE or PROPELLR sequence for e.g. They also learned a good deal from working with NYU's fastMRI

Re: AI models that predict disease are not as accurate as reports might suggest

#74

Post by drtgh »

spywaregorilla wrote: post #21
drtgh wrote: post #12

My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. An attempt to dispense with mathematicians to develop models. What amount of energy is necessary for an event to be reflected in a statistic? You have a box of 2x2 meters with ba…

> My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. Sure. Hardly controversial. > An attempt to dispense with mathematicians to develop models. What...? No. Definitely not. > What amount of energy is necessary for an event to be…

>I have no idea what this is saying. It sounds like you're shitting on statistics all of a sudden, which is weird, given that you seemed to favor mathematicians in the first part.

Mathematicians are specialized in problem solving, and as humans, their ability to predict and analyze data makes them more reliable developing models than a statistical equation. They have quite more tools than statistics one.

Someway, it is like if using the acronym AI to define statistical algorithms leads to a false sense of greater reliability than such human review, or even that it is not needed a deep human review. ML statistics takes algorithms out of the oven long before mathematicians does, at the expense of a big in accuracy difference.

The problem I think is people may take important decisions based in the result of such statistical algorithms without questioning

Re: AI models that predict disease are not as accurate as reports might suggest

#75

Post by drtgh »

dekhn wrote: post #40
drtgh wrote: post #12

My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. An attempt to dispense with mathematicians to develop models. What amount of energy is necessary for an event to be reflected in a statistic? You have a box of 2x2 meters with ba…

Statistics is not science- it's an application of probability theory and some other forms of math to hypothesis selection (among other things). It's scientific. We only use stats because that's the best method for dealing with imprecise and noisy data. Statistical thermodynamics contains all the necessary tools you need to answer your balls in a box question.

>Statistical thermodynamics contains all the necessary tools you need to answer your balls in a box question

The balls in a box example shows how ML statistics work. The string is adjustable, it can be adapted to different contours, but you have to discard data.

How do you compensate for the inclusion of data in the model without discarding others? The string has a limit in diameter by design, and you need to know the content of most of the data to make good decisions.

Re: AI models that predict disease are not as accurate as reports might suggest

#76

Post by spywaregorilla »

dirheist wrote: post #39
spywaregorilla wrote: post #22

Earlier quoted context omitted.

That's not a contradiction per se. It's easier to get spurriously high test scores with smaller datasets. It does not clearly demonstrate that the models are actually getting worse.

But if diagnosis are multimodal and rely upon large, multidimensional analysis of symptoms/bloodwork/past medical history, wouldn't adding more dimensions just increase dimensional sparsity and decrease the useful amount of conclusions you are able to draw from your variables? It's been a long time since I remember learning about the curse of dimensionality but if you increase the amount of datapoints you collect by…

I did mean samples (n size) not the number of features. But also, no your point isn't right. If you have a ton of variables, you'll be better able to overfit your models to a training set (which is bad). However, that's not to say that a fairly basic toolkit can't help you avoid doing that even with a ton of variables. What really matter is the effect size of the the variables you're adding. That is, whether or not they can actually help you predict the answer, distinctly from the other variables you have.

Stupid example: imagine trying to predict the answer of a function that is just the sum of 1,000,000 random variables. Obviously having all 1,000,000 variables will be helpful here, and the model will learn to sum them up.

In the real world, a lot of your variables either don't matter or are basically saying the same thing as some of your other variables so you don't actually get a lot of value from trying to expand your feature set mindlessly.

> if you increase the amount of datapoints you collect by half you would have to quadruple the amount of samples you have to retrieve any meaningful benefit, no?

I think you might be thinking about standard error. Where you divide the standard deviation of your data by sqrt of the number of samples. So quadrupling your sample size will cut the error in half?

Re: AI models that predict disease are not as accurate as reports might suggest

#77

Post by spywaregorilla »

drtgh wrote: post #74
spywaregorilla wrote: post #21

Earlier quoted context omitted.

> My humble opinion; AI is supposed to be the acronym for artificial intelligence, but marketing has usurped it to refer to machine learning, which is nothing more than a neo-language for defining statistical equations in a semi-automated way. Sure. Hardly controversial. > An attempt to dispense with mathematicians to develop models. What...? No. Definitely not. > What amount of energy is necessary for an event to be…

>I have no idea what this is saying. It sounds like you're shitting on statistics all of a sudden, which is weird, given that you seemed to favor mathematicians in the first part. Mathematicians are specialized in problem solving, and as humans, their ability to predict and analyze data makes them more reliable developing models than a statistical equation. They have quite more tools than statistics one. Someway, it…

I don't think most mathematicians have spent a great deal of time analyzing data tbh. Unless you mean statisticians.

Re: AI models that predict disease are not as accurate as reports might suggest

#78

Post by rscho »

tensor wrote: post #15

This is entirely unsurprising and has a very simple solution: keep adding more data. Our measurements of the accuracy of AI systems are only as good as the test data, and if the test data is too small, then the reported accuracies won't reflect the true accuracies of the model applied to wild data. Basically, we need an accurate measure of whether the test data set is statistically representative of wild data. In hea…

> This is entirely unsurprising and has a very simple solution: keep adding more data

Nope. Won't work. Biased data made bigger results only in bias confirmation. Which is the real problem.

Re: AI models that predict disease are not as accurate as reports might suggest

#79

Post by rvz »

Of course. No surprise there. Especially the ones made with 'Deep Learning'.

At this point, Each time AI and 'Deep Learning' is applied and then scrutinised, it almost always concludes and tends towards pure hype generated by investors and output garbage unexplainable results from broken models. The exact same goes for the self-driving scam.

'AI' is slowing starting to be getting outed as an exit scam.

Re: AI models that predict disease are not as accurate as reports might suggest

#80

Post by yrgulation »

bitL wrote: post #53

OK, AI is bad but compare it to human doctors/radiologists that are often worse. I still remember stats from some X-ray detection where AI diagnosed with 40% accuracy and the best human doctors with 38% accuracy (and median human doctors with 32% accuracy). Now what are we supposed to do?

Oh god (science for some of us) the same kind of logic for defending tesla’s fsd system. Both crappy and dangerous, but with cult like following.
Post reply on HN