Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FDA permits marketing of AI-based device to detect diabetes-related eye problems

fda.gov

Post reply on HN
Print view View original

41–50 of 104 posts

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#41

Post by j32fun »

bena wrote: post #39
j32fun wrote: post #32

Ultimately, you will need a doctor somewhere along the diagnostic process so that someone is there to assume liability for incorrect diagnoses.

What's crazy is that if AI is better than doctors by some significant degree, what do we do when the doctor and AI disagree? Like if doctors are right 85% of the time, but the AI is 90%. I guess we treat it as another doctor? Like if we have 4 opinions that agree, we go with that one regardless of the source of those opinions (as long as they meet some minimum competence threshold).

That's an interesting thought.

Currently, what happens is that if a diagnostic test comes back and it suggests something serious, say cancer, and the doctor does not pursue it, then the doctor would be liable if it did turn out to be cancer.

So if a machine disagreed with a doctor, then I would assume that the doctor will grudgingly have to investigate further until there is enough evidence to rule out that diagnosis.

#headache

What I can see happening is that patients will go to this machine for a second opinion. And if an opinion then returns that contradicts the primary physician, then an entire can of (legal) worms will be open.

--

Addendum:

To elaborate further, there is sometimes what's called the benefit of history.

Say a patient visits 10 doctors. The 10th doctor has an unfair advantage to the first 9 simply because he/she will have the prior knowledge of which diagnoses and treatments were incorrect.

Similarly for an AI vs Human Doctor situation, the incorporation of additional information (for the AI) would require considerable amount of big data to train in order to be able to recognize prior history, failed treatments, and such.

For image specific diagnoses (eg. recognizing melanoma, retinopathy), these do lend themselves to AI very nicely. For other diagnoses that contain a significant amount of, shall we say, "human factors", then less so.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#42

Post by kendallpark »

laythea wrote: post #36

I wonder if doctors will become like pilots? Ie. Almost redundant, but essential, at the same time.

As someone in this field (AI+medicine), I think this is the best analogy. Though a key distinction is that the human body is also a person whereas a plane is not. Physicians are there to promote the health of the person not just their mechanical parts. I've seen doctors fudge billing codes to help poor patients afford care. I've seen doctors pick up on domestic violence situations based on small social cues. There is a certain degree that healthcare relies on the humanness and empathy of physicians to promote human flourishing.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#43

Post by raverbashing »

asperous wrote: post #5

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

Any machine or specialist that wants to do a diagnostic based on a test result will have to consider other factors apart from the result.

The false positive bayesian math is a good illustrative example, but reality is more complicated. And no doctor will base their diagnostic solely on one number.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#44

Post by jhokanson »

killjoywashere wrote: post #3
ekianjo wrote: post #2

Does the FDA conduct code reviews? And how do they guarantee that the code or the training data does not change over time without them knowing?

They generally treat compute devices as black boxes. Performance is all that matters. But once an artifact is submitted, it's locked down. That and only that will be approved. You change a resistor, an if-then, it goes back to FDA.

What about learning? Would an update with more data and updated coefficients require going back to the FDA?

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#45

Post by laythea »

bobowzki wrote: post #40
laythea wrote: post #36

I wonder if doctors will become like pilots? Ie. Almost redundant, but essential, at the same time.

I'm a doctor and very much pro ML/AI. I'd love to have an autopilot I could watch in awe. Still a lot of practical tasks though that will be much harder to automate. And for the first few generation of AIs I guess someone will have to babysit them.

I would imagine having such an autopilot would make one less professed at any particular subject.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#46

Post by kendallpark »

For those interested in the research side of this, Google Brain actually published a study in JAMA on the same topic. They did clinical trials in India and should be publishing those results eventually.

https://static.googleusercontent.com/media/research.google.c...

In terms of how well the "experts" perform: "For moderate or worse DR, the majority decision of ophthalmologists had a sensitivity of 0.838 and specificity of 0.981"

http://www.aaojournal.org/article/S0161-6420(17)32698-2/abst...

And here's a video that describes what's going on in plain English:

https://www.youtube.com/watch?v=oOeZ7IgEN4o

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#47

Post by ujal »

"The 207,130 images collected were reduced to the 108,312 OCT images (from 4686 patients) and used for training the AI platform. Another subset of 633 patients not in the training set was collected based on a sample size requirement of 583 patients to detect sensitivity and specificity at 0.05 marginal error and 95% confidence. The test images (n = 1000) were used to evaluate model and human expert performance." --

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#48

Post by JumpCrisscross »

jdiez17 wrote: post #30
ekianjo wrote: post #2

Does the FDA conduct code reviews? And how do they guarantee that the code or the training data does not change over time without them knowing?

Generally, the FDA (or the government body responsible for certifying medical devices) does not conduct code reviews in the sense of looking at the code and trying to find bugs. The way it works is: the manufacturer of a medical device assesses the harm that can be caused by a software malfunction, and assigns it a safety classification (class A, B or C). Class A is used when no injury is possible, and class C is use…

> class C software it requires whitebox tests as well

What does this mean?

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#49

Post by epmaybe »

lallysingh wrote: post #33
epmaybe wrote: post #25

Earlier quoted context omitted.

Also keep in mind that if there are positives general practitioners would refer to a specialist anyways for treatment. These specialists would be more than equipped to detect false positives. Teleretina imaging is becoming more and more prevalent as well with eyePACS and Welch-Allyn having dedicated interpretation services, so patients wouldn't have to necessarily go somewhere for verification. I'm more worried about…

I guess the baseline for comparing the 13% would be the nonspecialist rate of the same.

Perhaps, that's one way to compare non-inferiority. Few if any primary care physicians take the time to look in ones eye (don't know how to use direct ophthalmoscope, don't have other specialized fundus cameras in clinic). Given that, if this tool forces them to take more retinal photographs of all patients, maybe we could detect diabetic disease before it is usually seen.

The standard practice today is that if a patient is determined to be diabetic then they get referred to an ophthalmologist visit once a year. In that case would comparing those rates of diagnosis be useful?

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#50

Post by asperous »

maxerickson wrote: post #19
asperous wrote: post #5

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

They are marketing the device strictly to make referrals to specialists. The FDA press release says as much anyway: https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/u... If the images are of sufficient quality, the software provides the doctor with one of two results: (1) “more than mild diabetic retinopathy detected: refer to an eye care professional” or (2) “negative for more than mild diabetic retinopath…

Yeah from the article:

> IDx-DR founder Michael Abràmoff told Science News. “It makes the clinical decision on its own.”

But I would guess a specialist would still need to be involved since it's not a fool-proof system. A specialist might take other symptoms or variables into account when making the diagnosis or order further tests. While this tool might be useful for blanket screening considering that it is harmless, it seems like it's hardly going to be "making the decision on its own" and prescribing treatment.

Post reply on HN