I’m a radiologist, and I’ve worked on these sorts of research studies before. You absolutely do need an IRB to do deep learning on clinical MRIs. I’ve had to write IRBs just to collect retrospective MRIs. Even for studies that are potentially IRB exempt, going through the exemption request takes hours of work on my part and weeks to months before the IRB grants the exception.
As other commenters mention, the “theoretical risk to a human” encompasses nearly all research. For what I do, any imaging studies that aren’t de-identified pose a theoretical risk to patient privacy. If you try to de-identify images, you learn that this is nearly impossible. Sure, you can try your best to scrub DICOM headers, but these headers are mis-used by vendors, so identifying information can appear almost anywhere. You could delete the headers entirely, but then you lose a lot of metadata that you may need to properly display the images. Further, people contend that you can identify individual peoples’ faces if you 3D-reconstruct CT/MR images, so then you have to expend resources to delete faces from all head/neck/brain imaging. Confirming that this was done properly requires manual review and limits the size of your dataset.
Edit: I think the disagreements here are partly due to institutional differences in IRB requirements and partly due to conflating “IRB exemptions” with the idea of not having to interact with the IRB at all. You always have to interact with the IRB—even just to obtain an exemption. While obtaining IRB approval is a cumbersome process, obtaining IRB exemption is only slightly less cumbersome. I’m sure this varies across institutions, but I’ve been at three different large urban academic centers, and obtaining exemptions has been a multi-month process at all three.