The best story I heard about machine learning and radiology was when folks were racing to try to detect COVID in lung X-rays. As I recall, one group had fairly good success, but eventually someone figured out that their data set had images from a low-COVID hospital and a high-COVID hospital, and the lettering on the images used different fonts. The ML model was detecting the font, not the COVID. [a bit of googling la…
Sorry, but in the absence of general limiting principles that rule out such a scenario, that's how it's going to shake out. Visual models are too good at exactly this type of work.