MindEye2: Shared-Subject Models Enable fMRI-to-Image with 1 Hour of Data

Impressive

Looking at the result (figure 4), the model looks more like a good classifier than an image retriever, the generated images have the same subject as the original but are very different, even the colors of the subjects.

It's remarkable that this works, and it's equally remarkable that the authors succeeded using a (deliberately) naive approach with the fMRI data. I was just teaching the visual system to students over the last few weeks: cognitive neuroscience/neuropsychology can tell us a lot about what brain systems should be involved here. It's such a tempting research domain to dive into using a more sophisticated approach, but then again, maybe the authors' simpler data-driven method is a strength.

From the article,

> The present work demonstrates that it is now practical for patients to undergo a single MRI scanning session and produce enough data to perform high-quality reconstructions of their visual perception.

> Such image reconstructions from brain activity are expected to be systematically distorted due to factors including mental state, neurological conditions, etc.

> This could potentially enable novel clinical diagnosis and assessment approaches, including applications for improved locked-in (pseudocoma) patient communication (Monti et al., 2010) and brain-computer interfaces if adapted to real-time analysis (Wallace et al., 2022) or non-fMRI neu-roimaging modalities.

> As technology continues to improve, we note it is important that brain data be carefully protected and companies collecting such data be transparent with their use.

So here is a question: what would happen if this process faced a person who was specifically hostile to it? Could you "trick" the process by intensely thinking of, for instance, a yellow ball while viewing a cat picture, as one example?