Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FDA permits marketing of AI-based device to detect diabetes-related eye problems

fda.gov

Post reply on HN
Print view View original

11–20 of 104 posts

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#11

Post by jforman »

jadedhacker wrote: post #7

I wonder what factors made this decision possible? I love the idea of automated diagnosis but the performance rates are 87% true positive and 90% true negative in the article. Seems a bit low. Maybe people aren't getting diagnosed at very high rates? That would be a reasonable justification for deployment with somewhat less than perfect accuracy. Anyone have any insight?

For new stuff, FDA performs a holistic cost/benefit analysis.

In this case, they might weigh: * How many new cases are caught by expanding access to specialist tools * What fail safes exist in current course of care — how does a false negative result in a worse outcome for a patient than if they had had no diagnostic at all * etc.

The summary of their decision is public record, but not the detailed analysis.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#12

Post by denzil_correa »

> But of course, not having a specialist “looking over the shoulder,” as Abràmoff puts it, raises the question of who will be responsible when the diagnosis is wrong

Ultimately, accountability and transparency will be the Achilles heel.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#14

Post by dboreham »

TaylorAlexander wrote: post #4

I feel like this has the same promise as self driving cars - raise the floor for the quality of service while also experiencing random unexplained failures that its human supervisors fail to notice in time.

Based on my experience watching doctors (good ones) trying to diagnose "weird things", they are just manually executing an expert system algorithm anyway. They aren't doing what an engineer might expect -- working from a basic understanding of how the body works and looking for plausible explanations. They're instead simply pattern matching against a database of facts.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#15

Post by Eridrus »

asperous wrote: post #5

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

I think the number is closer to 6%.

90% of people are accurately detected as not having the disease, i.e. 10% FP rate. So ~3m people would be falsely diagnosed as having the complication, to 200k/3m have the disease.

10% isn't a great number, but it isn't clear from this coverage whether this complication is generally asymptomatic or not. If there are symptoms to go with it, the numbers may be far better.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#16

Post by monkeynotes »

denzil_correa wrote: post #12

> But of course, not having a specialist “looking over the shoulder,” as Abràmoff puts it, raises the question of who will be responsible when the diagnosis is wrong Ultimately, accountability and transparency will be the Achilles heel.

Ultimately it has to be the company providing the technology that is liable, but I bet you they have a terms of use clause that puts the responsibility on the clinics using the technology.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#17

Post by hanklazard »

jadedhacker wrote: post #7

I wonder what factors made this decision possible? I love the idea of automated diagnosis but the performance rates are 87% true positive and 90% true negative in the article. Seems a bit low. Maybe people aren't getting diagnosed at very high rates? That would be a reasonable justification for deployment with somewhat less than perfect accuracy. Anyone have any insight?

It also seems like a quick way to screen for this condition, vs the level of concern that a primary care doc might need to have before referring someone to an ophthalmologist. In other words, you could screen more people, even those that are asymptomatic (but have diabetes) and potentially catch retinal disease earlier. Even without mind-blowing sensitivity / specificity, this could preserve a lot of people’s vision.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#18

Post by monkeynotes »

rhacker wrote: post #6
ekianjo wrote: post #2

Does the FDA conduct code reviews? And how do they guarantee that the code or the training data does not change over time without them knowing?

Seems like they are using this to get a proper referral to a specialist rather than using this as sole diagnosis. The code itself is probably 15 lines using Tensorflow or other framework I'm guessing, but could be wrong.

If it is that simple I am wasting my time in my current job.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#19

Post by maxerickson »

asperous wrote: post #5

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

They are marketing the device strictly to make referrals to specialists. The FDA press release says as much anyway:

https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/u...

If the images are of sufficient quality, the software provides the doctor with one of two results: (1) “more than mild diabetic retinopathy detected: refer to an eye care professional” or (2) “negative for more than mild diabetic retinopathy; rescreen in 12 months.” If a positive result is detected, patients should see an eye care provider for further diagnostic evaluation and possible treatment as soon as possible.

Are you interpreting the Verge or talking about some other statement?

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#20

Post by jpredham »

There are several comments about the accuracy of the algorithm, but Doctors also struggle with diagnosis. In the Deep Mind study on DR they found that for about 20% of referable cases doctors disagreed on the diagnosis about 40-50% of the time. In order to combat this they had at least 3 and up to 7 opthamologists grade each image.

Source: https://jamanetwork.com/journals/jama/fullarticle/2588763

Post reply on HN