Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FDA permits marketing of AI-based device to detect diabetes-related eye problems

fda.gov

Post reply on HN
Print view View original

81–90 of 104 posts

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#81

Post by return1 »

ksolanki wrote: post #74

As a founder of another company in this field, let me start by saying that this approval is a big deal. Kudos to IDx. This is the very first time FDA has approved a fully automated CADx (computer aided diagnosis) device. Eyenuk is also on it's way to an FDA approval and it is a lot of work conducting the prospective clinical trials. There are some misconceptions on the thread, so let me help clear them up. A screenin…

what is the screening like? is it possible there will be automated self-service 'kiosks' for that?

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#83

Post by YeGoblynQueenne »

ksolanki wrote: post #75
YeGoblynQueenne wrote: post #60

>> In one clinical trial that used more than 900 images, IDx-DR correctly detected retinopathy about 87 percent of the time, and could correctly identify those who didn’t have the disease about 90 percent of the time. I read that as .87 accuracy, .9 specificity (True Negative Rate). However, I can't find in the link provided in the article above the sensitivity (recall or True Positive Rate). I'm guessing it goes a b…

This is not completely correct. Sensitivity (87% in this case) is not same as accuracy, nor is specificity (89.5% in this case) same as true negative rate. Sensitivity generally gives an indication of safety of a medical device, and specificity gives a general indication of effectiveness.

The link changed since I posted my comment :)

The original article on Verge reported "87% accuracy" and 90% what sounded like TNR. The new link points to an FDA page that makes it more likely that ".87" is actually sensitivity:

Dx-DR was able to correctly identify the presence of more than mild diabetic retinopathy 87.4 percent of the time and was able to correctly identify those patients who did not have more than mild diabetic retinopathy 89.5 percent of the time.

So, I guess, something like this:

           Predicted + Predicted - Total
  Actual + 393         57          450
  Actual - 47          403         450
  -------------------------------------
  Total    440         460         900
  
  Accuracy:             0.8844
  Error:                0.1156
  True Positive Rate:   0.8733
  True Negative Rate:   0.8956
  Precision:            0.8932
  Recall (TPR):         0.8733
  F-Score:              0.8831
Closest I can get with exactly 900 cases :0

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#84

Post by drta »

ksolanki wrote: post #74

As a founder of another company in this field, let me start by saying that this approval is a big deal. Kudos to IDx. This is the very first time FDA has approved a fully automated CADx (computer aided diagnosis) device. Eyenuk is also on it's way to an FDA approval and it is a lot of work conducting the prospective clinical trials. There are some misconceptions on the thread, so let me help clear them up. A screenin…

> DR is the leading cause of blindness in working-age adults even in the developed world.

It is changing though - it's not been true for the UK since 2014 - https://www.gov.uk/government/news/diabetes-no-longer-leadin...

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#85

Post by dboreham »

nradov wrote: post #78
dboreham wrote: post #14

Earlier quoted context omitted.

Based on my experience watching doctors (good ones) trying to diagnose "weird things", they are just manually executing an expert system algorithm anyway. They aren't doing what an engineer might expect -- working from a basic understanding of how the body works and looking for plausible explanations. They're instead simply pattern matching against a database of facts.

A lot of diagnostic skill in complex cases is based on clinical intuition developed over many years of practice. That's qualitatively different from an expert system executing defined rules against facts.

You'd think, but honestly in my experience it wasn't like that. An expert system would have done as well, or better. Possibly my experiences were colored by the cross-specialization nature of the issue -- specialists seem reluctant to engage any thinking outside of their area, I found. Like a software engineer would never consider that the problem they're investigating is caused by memory bits flipping randomly.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#86

Post by drta »

pontus wrote: post #64
asperous wrote: post #5

The company is saying you don't need a specialist, but after bayes theorem (using 90%TN 87%TP and D(A)= 200,000 complication / 29,100,000 diabetes), the chance you have this condition after the machine says you do is 0.83%.

Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease. As a result, the prior of 200k / 29M is not quite right and I'm guessing that the true prior is likely much higher.

>Keep in mind that the test wouldn't be administered on any randomly chosen diabetic patient but presumably rather on those who already exhibit some type of vision imparement consistent with the disease.

Not true. The aim is to treat patients before they become symptomatic. Outcomes are much worse otherwise.

2min promo on Diabetic Eye Screening: https://www.youtube.com/watch?v=PK1Y-1BKFn0

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#87

Post by homero »

return1 wrote: post #81
ksolanki wrote: post #74

As a founder of another company in this field, let me start by saying that this approval is a big deal. Kudos to IDx. This is the very first time FDA has approved a fully automated CADx (computer aided diagnosis) device. Eyenuk is also on it's way to an FDA approval and it is a lot of work conducting the prospective clinical trials. There are some misconceptions on the thread, so let me help clear them up. A screenin…

what is the screening like? is it possible there will be automated self-service 'kiosks' for that?

Since it's not dilated, it's possible

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#88

Post by Maybestring »

bena wrote: post #39
j32fun wrote: post #32

Ultimately, you will need a doctor somewhere along the diagnostic process so that someone is there to assume liability for incorrect diagnoses.

What's crazy is that if AI is better than doctors by some significant degree, what do we do when the doctor and AI disagree? Like if doctors are right 85% of the time, but the AI is 90%. I guess we treat it as another doctor? Like if we have 4 opinions that agree, we go with that one regardless of the source of those opinions (as long as they meet some minimum competence threshold).

The doctor isn't going to be making decisions without access to the computer diagnosis.

The computer aided diagnosis isn't another doctor, it's another stethoscope.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#89

Post by Ensorceled »

jadedhacker wrote: post #7

I wonder what factors made this decision possible? I love the idea of automated diagnosis but the performance rates are 87% true positive and 90% true negative in the article. Seems a bit low. Maybe people aren't getting diagnosed at very high rates? That would be a reasonable justification for deployment with somewhat less than perfect accuracy. Anyone have any insight?

I used to work in medical startups and twice we were approved for simply being better than standard of care.

If you can show a statistically better chance of a good outcome with a small chance of significantly worse outcome, the FDA will often approve.

Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems

#90

Post by ksolanki »

drta wrote: post #84
ksolanki wrote: post #74

As a founder of another company in this field, let me start by saying that this approval is a big deal. Kudos to IDx. This is the very first time FDA has approved a fully automated CADx (computer aided diagnosis) device. Eyenuk is also on it's way to an FDA approval and it is a lot of work conducting the prospective clinical trials. There are some misconceptions on the thread, so let me help clear them up. A screenin…

> DR is the leading cause of blindness in working-age adults even in the developed world. It is changing though - it's not been true for the UK since 2014 - https://www.gov.uk/government/news/diabetes-no-longer-leadin...

That's correct. UK is the only major (in some vague sense) Country where diabetic retinopathy is not the leading cause of preventable blindness in adults. This is very likely because they are able to screen more than 80% (nearing 85%) of their diabetic population, an impressive feat. This leads to another issue: they need to consistently grade the retinal images of over 2.2 million patients with diabetes. This is where AI could help -- in improving the consistency and turn-around time and we are working with the NHS UK to explore this.
Post reply on HN