Machine learning is booming in medicine, but also facing a credibility crisis • Hacker News Board

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#61

Post by gjulianm » Tue, Jun 08, 2021, 12:52 PM UTC

klodolph wrote: post #54

gjulianm wrote: post #51

Earlier quoted context omitted.

See my comment below for the same discussion: https://news.ycombinator.com/item?id=27433368 My point isn't that you need 100% accuracy. It's that a diagnosis is a process, not a single test. If your model applies to a single step of the process, and if it doesn't remove enough uncertainty about that step, you're still going to continue the diagnostic process and the model is not going to change anything really.

You said ML will have “no effect on those procedures unless it's 100% accurate.” I disagree with that, and think it’s a bit absurd—the 100% figure is not possible anyways, so you might as well stick some other impossible predicate in there and the statement is equally insightful. E.g., “it will have no effect on outcomes until pigs fly.” There will always be uncertainty, and uncertainty isn’t the only relevant parame…

> There will always be uncertainty, and uncertainty isn’t the only relevant parameter.

But it's a really important one and a lot of medical ML research doesn't seem to address in the proper sense. A very simple example: a ML model that classifies lung nodules as benign or malign, with 95% accuracy vs 70% accuracy of regular radiologists. Very good, right? But for actual, real world results, you need to see how the patient outcomes change. If the patients where the model and doctors disagree were going to have extra tests or followup regardless of what the model says, the model is not actually offering anything new despite the increase in accuracy.

So no, it's not absurd to say that models that work for a single type of test need to be very, very accurate to actually bring changes to the procedures that are worth the investment. What matters is whether they actually change patient outcomes, and sometimes it seems like the ML researchers barely consider it.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#62

Post by Enginerrrd » Tue, Jun 08, 2021, 1:05 PM UTC

rscho wrote: post #47

gjulianm wrote: post #8

Some radiologists think that AI will be really good for filtering out normal images, so that they only have to review anomalies. But I don't think that a model that detects diseases will be too successful, even if they manage to make it really work. For one, it's actually difficult to interpret and find signs in radiologic images. Obvious signs are obvious, but there are others that could be image artifacts, or just…

> modern machines can detect anomalies but doctors still learn how to interpret EKGs and double check what the machine says. Uh, TBH no professional so much as glances through the automated EKG summary. It's utterly useless and could be deleted with zero consequences.

I've seen a lot of ECG results, and I always did my own interpretation, but I always found the machine to be accurate, which is unsurprising since distinguishing rhythms is very codified anyway. My only issue with the machine is that it's not really doing something difficult that needs doing. You still need someone that understands how to read ECG's and make decisions about patient management based on past and present conditions. That is the hard part.

I'm curious though. Can you elaborate on the types of issues you have encountered? What brand of equipment were you using?

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#63

Post by Enginerrrd » Tue, Jun 08, 2021, 1:08 PM UTC

gjulianm wrote: post #53

exporectomy wrote: post #52

Earlier quoted context omitted.

I was thinking more like ultrasound. If some crude scanner was cheap enough for every GP's office to have, and an AI could decipher the noisy signal which is perhaps too messy for a human to read, it could be used routinely like blood pressure measurement and stethoscopes are. If it's a regular screening test rather than a response to the patient's complaint, then false negatives wouldn't be as much of a problem beca…

Ultrasound scanners are fairly cheap. But operating them is not easy, you still need training to use them and know what you're looking at. > it's better than nothing. Not necessarily, that's the issue with screening asymptomatic people. You have to balance the consequences and rates of false positives with the benefits of true positives. If ultrasound screening mostly catches indolent diseases, or those where catchin…

The potential harm runs deeper than this even. Once a structural abnormality is found, the nocebo effect can come into play and essentially cause symptoms in asymptomatic people, which is a terrible result.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#64

Post by itsoktocry » Tue, Jun 08, 2021, 1:12 PM UTC

bayesian_horse wrote: post #44

axelroze wrote: post #30

Earlier quoted context omitted.

A friend of mine once got rejected from a big name company after a few months of interviews. Reason was that he did not have a PhD. It's hard to get into the business. You could try to start-up and then plan to get acquihired. Or go and do a PhD. It's never too late for more schooling.

I'm not getting into any PhD programs though, because my CV is essentially radioactive (I tried...). I'm not getting any funding for a startup, for much of the same reasons.

>I'm not getting any funding for a startup, for much of the same reasons.

I hate to ask why you believe this, because I see VC money being thrown at borderline low-lifes with mediocre ideas. There are so many "angel investors" on Twitter of companies I've never heard of...

The SV bubble can warp your brain, but an outsider's tip is to focus on your idea before you start worrying about funding it...

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#65

Post by itsoktocry » Tue, Jun 08, 2021, 1:15 PM UTC

throwaways885 wrote: post #40

ecmascript wrote: post #15

Earlier quoted context omitted.

Why not build something on your spare time and try to create your own job then?

How would one person bootstrap a venture in medical ML without initial funding or datasets?

Do you think it's the model and the data that's novel? That's exactly why AI is failing.

How about starting with a problem to solve? You don't need anything but a pen and paper. Gotta start somewhere.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#66

Post by inglor_cz » Tue, Jun 08, 2021, 1:22 PM UTC

Enginerrrd wrote: post #63

gjulianm wrote: post #53

Earlier quoted context omitted.

Ultrasound scanners are fairly cheap. But operating them is not easy, you still need training to use them and know what you're looking at. > it's better than nothing. Not necessarily, that's the issue with screening asymptomatic people. You have to balance the consequences and rates of false positives with the benefits of true positives. If ultrasound screening mostly catches indolent diseases, or those where catchin…

The potential harm runs deeper than this even. Once a structural abnormality is found, the nocebo effect can come into play and essentially cause symptoms in asymptomatic people, which is a terrible result.

Also, one thing that is found does not preclude existence of other things to be found. The first discovered problem may attract so much attention that potential others are neglected.

Tl;dr: the first diagnosis tends to stick with the patient.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#67

Post by inglor_cz » Tue, Jun 08, 2021, 1:24 PM UTC

ThomPete wrote: post #38

ChuckNorris89 wrote: post #28

Earlier quoted context omitted.

Just because China hoards more data doesn't guarantee any success. If just data collection would be enough then all crime should have been eradicated in the US by how much data NSA has.

It guarantees more access to training data which is crucial for development of any superiority when it comes to applied AI. The US still have a better academic culture but the chinese are catching up. Nothing is inevitable.

The strength of the US was always in the attractivity of the country for talented foreigners.

Is the same true of China, or will China have to rely on its homegrown cadre of scientists? Granted, there might be a lot of talent in a population of 1,4 billion.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#68

Post by randomopining » Tue, Jun 08, 2021, 1:35 PM UTC

spoonjim wrote: post #20

> By far the biggest problem — and the trickiest to solve — points to machine learning’s Catch-22: There are few large, diverse data sets to train and validate a new tool on, and many of those that do exist are kept confidential for legal or business reasons. This is why China will win the AI Age.

So China's (1.4 bil and dropping) smart people vs the rest of the world's smart people (6-7 bil). And China doesn't get easy stepping stones when it wants by stealing secrets from western companies, uni's, and the us military.

Lets go.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#69

Post by azalemeth » Tue, Jun 08, 2021, 1:36 PM UTC

hdhjebebeb wrote: post #56

exporectomy wrote: post #52

Earlier quoted context omitted.

I was thinking more like ultrasound. If some crude scanner was cheap enough for every GP's office to have, and an AI could decipher the noisy signal which is perhaps too messy for a human to read, it could be used routinely like blood pressure measurement and stethoscopes are. If it's a regular screening test rather than a response to the patient's complaint, then false negatives wouldn't be as much of a problem beca…

I had an abdominal ultrasound recently - the actual process of placing the probe and annotating the images seems like a pretty specific technical skill. I don't think any old person (or even a doctor) off the street can just rub the wand on you and find your kidneys for instance.

Indeed. US is a highly technical, specific skill. Even within a specific use case -- e.g. echocardiography -- there's a specific series of qualifications that cardiologists need to get in order to be deemed competent. The rotation of your hand in a specific position in transthoracic echocardiography can change the resulting ejection fraction of the heart -- the main functional outcome. It's a simple thing to see done; quite another for you to do yourself and make safe decisions.

Re: Machine learning is booming in medicine, but also facing a credibility crisis

#70

Post by PeterisP » Tue, Jun 08, 2021, 2:09 PM UTC

BiteCode_dev wrote: post #10

Unless the AI can talk tot the patient to get some context, it's going to be taking decisions with only very partial information, no matter how good it is. Still, I'm thinking that as it improves, it's going to show that doctors are not that good at their job on average, and that's going to be fun to watch.

Radiologists aren't talking to the patient either and don't have earlier scans available, they "just" comment on the image; so the doctor who does have the patient, their history and the radiology report(s) might as well receive the report from a ML system instead of a human radiologist.