Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.
People with disabilities could benefit greatly from this.
fMRI-to-image with contrastive learning and diffusion priors
11–20 of 69 posts
Re: fMRI-to-image with contrastive learning and diffusion priors
#12Human communication will change dramatically once useful invasive brain-computer interfaces are available. People will suddenly realize that the reason language is primarily serial is simply due to the fact that it must be conveyed by a series of sounds. There will likely be a new type of visual language used via BCI "telepathy". It may have some ordering but will not rely so heavily on serializing information, since…
In a popular sci-fi (avoiding spoliers), the alien race has transparent skulls, and their visible thoughts are broadcast to anyone within visual range. It does seem more efficient than sound.
You may base64 to spoiler proof it
Re: fMRI-to-image with contrastive learning and diffusion priors
#13Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.
Re: fMRI-to-image with contrastive learning and diffusion priors
#14Earlier quoted context omitted.
In a popular sci-fi (avoiding spoliers), the alien race has transparent skulls, and their visible thoughts are broadcast to anyone within visual range. It does seem more efficient than sound.
I want to know the scifi You may base64 to spoiler proof it
Re: fMRI-to-image with contrastive learning and diffusion priors
#15This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.
Re: fMRI-to-image with contrastive learning and diffusion priors
#16> I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you'd get an overfitted picture of another teddy bear from the training dataset instead.
> The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by 'mind reading' subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image.
> If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it's entirely cherry-picked coincidences.
> 1. https://i.imgur.com/ILCD2Mu.png
Re: fMRI-to-image with contrastive learning and diffusion priors
#17Here's a radiograph of the primary visual cortex created in 1982 by projecting a pattern onto a macaque's retina: https://web.archive.org/web/20100814085656im_/http://hubel.m...
An injection of radioactive sugar lets you see where the neurons were firing away and metabolizing the sugar.
Re: fMRI-to-image with contrastive learning and diffusion priors
#18Human communication will change dramatically once useful invasive brain-computer interfaces are available. People will suddenly realize that the reason language is primarily serial is simply due to the fact that it must be conveyed by a series of sounds. There will likely be a new type of visual language used via BCI "telepathy". It may have some ordering but will not rely so heavily on serializing information, since…
Something else that's interesting about language is it's just a form of compressive medium for thoughts; I think of a concept, then I put it into words (compression) that you the listener then have to interpret and understand (decompression) and then fit your brain state to the new data you've received. It's overall a very lossy medium compared to what brains can do. It would be much easier to beam my thoughts and images and videos in my mind directly to you.
Unless you or I have aphantasia, of course.
Re: fMRI-to-image with contrastive learning and diffusion priors
#19Earlier quoted context omitted.
I want to know the scifi You may base64 to spoiler proof it
I think it's "VGhlIFRocmVlLUJvZHkgUHJvYmxlbQ==" The aliens cannot lie to each other (they don't even have the idea of a lie), because their thoughts are transparent to each other.