Wasn't a similar claim made about an AI detecting skin cancer from moles? Once the AI was deployed in the real world is failed miserably. I think it was a ton of false-positives because it was trained on images where cancerous moles all had images of rulers with them and the benign ones didn't have rulers in the image. So it just picked up on the ruler as a cancer indicator.
Would you happen to have a source for that story? My workplace has really swallowed the AI Kool-Aid lately, so I would like to have some cautionary counterexamples to demonstrate potential pitfalls of the technology. It's got a lot of interesting applications for our field which I am excited about, but there seems to be a tendency among non-experts to consider it a magic bullet that can solve any sort of problem. In…
From the abstract:
This study’s findings suggest that skin markings significantly interfered with the CNN’s correct diagnosis of nevi by increasing the melanoma probability scores and consequently the false-positive rate. A predominance of skin markings in melanoma training images may have induced the CNN’s association of markings with a melanoma diagnosis. Accordingly, these findings suggest that skin markings should be avoided in dermoscopic images intended for analysis by a CNN.