Since a lot of the discussion was objecting to the title and/or exaggerated coverage, and the press release is more factual (let that sink in for a second), we changed the URL from https://www.theverge.com/2018/4/11/17224984/artificial-intel... to the press release.
FDA permits marketing of AI-based device to detect diabetes-related eye problems
71–80 of 104 posts
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#72>> In one clinical trial that used more than 900 images, IDx-DR correctly detected retinopathy about 87 percent of the time, and could correctly identify those who didn’t have the disease about 90 percent of the time. I read that as .87 accuracy, .9 specificity (True Negative Rate). However, I can't find in the link provided in the article above the sensitivity (recall or True Positive Rate). I'm guessing it goes a b…
"For moderate or worse DR, the majority decision of ophthalmologists had a sensitivity of 0.838 and specificity of 0.981"
Note that this is "majority decision" of multiple ophthalmologists, which is going to be better than the average ophthalmologist's single diagnosis.
http://www.aaojournal.org/article/S0161-6420(17)32698-2/abst...
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#73Earlier quoted context omitted.
I'm a doctor and very much pro ML/AI. I'd love to have an autopilot I could watch in awe. Still a lot of practical tasks though that will be much harder to automate. And for the first few generation of AIs I guess someone will have to babysit them.
Ah, but who's going to treat the next few generations of AI, when there are no more doctors left who can "fly" without their "autopilot"?
I think a lot of commercial pilots would take offense to this. Commercial pilots can still fly without autopilot. Just as much as you can drive on the highway without cruise control.
https://pilotjobs.atpflightschool.com/2016/07/27/how-much-do...
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#74As a founder of another company in this field, let me start by saying that this approval is a big deal. Kudos to IDx. This is the very first time FDA has approved a fully automated CADx (computer aided diagnosis) device. Eyenuk is also on it's way to an FDA approval and it is a lot of work conducting the prospective clinical trials.
There are some misconceptions on the thread, so let me help clear them up. A screening test is indicated on an annual basis for anyone with diabetes who does NOT have visual symptoms. Diabetic retinopathy (DR) progresses without any symptoms and is preventable if detected early, but despite its preventable nature, DR is the leading cause of blindness in working-age adults even in the developed world.
The test is for screening rather than providing a full diagnosis and is not intended to replace a dilated ophthalmologist examination. You don't need a specialist to screen, but you need a specialist to diagnose and treat. Sensitivity implies the percentage of times the test is able to correctly identify the presence of more than mild diabetic retinopathy (in this case, 87.4 percent of the time) and specificity is the percentage of times the test was able to correctly identify those patients who did not have more than mild diabetic retinopathy (in this case, 89.5 percent of the time). Note that neither sensitivity nor specificity implies accuracy. The sensitivity and specificity generally compare well to that achieved by humans.
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#75>> In one clinical trial that used more than 900 images, IDx-DR correctly detected retinopathy about 87 percent of the time, and could correctly identify those who didn’t have the disease about 90 percent of the time. I read that as .87 accuracy, .9 specificity (True Negative Rate). However, I can't find in the link provided in the article above the sensitivity (recall or True Positive Rate). I'm guessing it goes a b…
This is not completely correct. Sensitivity (87% in this case) is not same as accuracy, nor is specificity (89.5% in this case) same as true negative rate.
Sensitivity generally gives an indication of safety of a medical device, and specificity gives a general indication of effectiveness.
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#76> But of course, not having a specialist “looking over the shoulder,” as Abràmoff puts it, raises the question of who will be responsible when the diagnosis is wrong Ultimately, accountability and transparency will be the Achilles heel.
Ultimately it has to be the company providing the technology that is liable, but I bet you they have a terms of use clause that puts the responsibility on the clinics using the technology.
The more important question is what they're actually held to. In the case of a diagnostic test like this, the most reasonable thing is to hold them to the false negative and false positive rates that they say the product has. No real-world diagnostic test is perfect. Instead, we design the systems around the tests to accommodate the fact that they do have errors. As long as the device manufacturer is correct about their probabilities of errors, then we can incorporate them into a system that works better than what we had before. Or we can know that we cannot use them in a useful way, and just ignore it.
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#77I think the title is fine, but a lot of the comments are applying what the software does to medicine in general. What this software is used for is very specific, but also very useful in that it is a common medical problem. It is used only to help diagnose diabetic retinopathy (ie eye damage caused by diabetes). This is AI Vision software used to analyze a photograph of someone's retina to detect damage. In essence it…
Eric Schmidt (former Google chairman) proposed building an AI scribe during his HIMSS 2018 conference keynote address. He claimed it should be possible within 10 years. I hope he's right, but I suspect he underestimates the difficulty of building reliable clinical systems.
http://www.himssconference.org/education/sessions/keynote-sp...
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#78I feel like this has the same promise as self driving cars - raise the floor for the quality of service while also experiencing random unexplained failures that its human supervisors fail to notice in time.
Based on my experience watching doctors (good ones) trying to diagnose "weird things", they are just manually executing an expert system algorithm anyway. They aren't doing what an engineer might expect -- working from a basic understanding of how the body works and looking for plausible explanations. They're instead simply pattern matching against a database of facts.
A lot of diagnostic skill in complex cases is based on clinical intuition developed over many years of practice. That's qualitatively different from an expert system executing defined rules against facts.
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#79Ultimately, you will need a doctor somewhere along the diagnostic process so that someone is there to assume liability for incorrect diagnoses.
What's crazy is that if AI is better than doctors by some significant degree, what do we do when the doctor and AI disagree? Like if doctors are right 85% of the time, but the AI is 90%. I guess we treat it as another doctor? Like if we have 4 opinions that agree, we go with that one regardless of the source of those opinions (as long as they meet some minimum competence threshold).
I know that this is a totally whack-job comment, but the TV show "the Good Doctor" is kinda leaning this way. Instead of relying on a ton of personal bias - the main character is generally more similar to how a ML would diagnose things, obviously there's no way to establish any foundation to this since it's based on a TV show. But it offers a vision of what you're saying but instead of AI it's a Savant syndrome individual that is making better judgments than the rest of the doctors. That being said, I would imagine a Savant is being placed in a role like that is less likely than the show portrays so where does that place AI?
Re: FDA permits marketing of AI-based device to detect diabetes-related eye problems
#80Ultimately, you will need a doctor somewhere along the diagnostic process so that someone is there to assume liability for incorrect diagnoses.