Live data from Hacker News

fMRI-to-image with contrastive learning and diffusion priors

stability.ai

11–20 of 69 posts

Re: fMRI-to-image with contrastive learning and diffusion priors

#11
post #3

Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.

People with disabilities could benefit greatly from this.

As long as they want to talk about london buses, steam trains, surfing and football.

Re: fMRI-to-image with contrastive learning and diffusion priors

#12
post #10
post #6

Human communication will change dramatically once useful invasive brain-computer interfaces are available. People will suddenly realize that the reason language is primarily serial is simply due to the fact that it must be conveyed by a series of sounds. There will likely be a new type of visual language used via BCI "telepathy". It may have some ordering but will not rely so heavily on serializing information, since…

In a popular sci-fi (avoiding spoliers), the alien race has transparent skulls, and their visible thoughts are broadcast to anyone within visual range. It does seem more efficient than sound.

I want to know the scifi

You may base64 to spoiler proof it

Re: fMRI-to-image with contrastive learning and diffusion priors

#13
post #3

Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.

Yeah, the first thing that comes to mind(har har) when I see this is that we'd be better off trying to develop better scanning technology. You can't exactly walk around town with an MRI strapped to your skull.

Re: fMRI-to-image with contrastive learning and diffusion priors

#14
post #10

Earlier quoted context omitted.

In a popular sci-fi (avoiding spoliers), the alien race has transparent skulls, and their visible thoughts are broadcast to anyone within visual range. It does seem more efficient than sound.

I want to know the scifi You may base64 to spoiler proof it

I think it's "VGhlIFRocmVlLUJvZHkgUHJvYmxlbQ==" The aliens cannot lie to each other (they don't even have the idea of a lie), because their thoughts are transparent to each other.

Re: fMRI-to-image with contrastive learning and diffusion priors

#15
I think the method of merging the pipelines via img2img should use controlnet. Possibly needing to be finetuned specifically for this, although existing controlnet models might work fine for this.

This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.

Re: fMRI-to-image with contrastive learning and diffusion priors

#16
Wasn't there something similar a few months ago on HN and where the top comment talked about how it's not as impressive as it sounds [0]? The main issue is that this type of methodology is pulling from a pool of images, not literally reconstructing what image was seen in the brain directly.

> I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you'd get an overfitted picture of another teddy bear from the training dataset instead.

> The best evidence for this is a picture(1) from page 6 of the paper. Look at the second row. The building generated by 'mind reading' subject 2 and 4 look strikingly similar, but not very similar to the ground truth! From manually combing through the training dataset, I found a picture of a building that does look like that, and by scaling it down and cropping it exactly in the middle, it overlays rather closely(2) on the output that was ostensibly generated for an unrelated image.

> If so, at most they found that looking at similar subjects light up similar regions of the brain, putting Stable Diffusion on top of it serves no purpose. At worst it's entirely cherry-picked coincidences.

> 1. https://i.imgur.com/ILCD2Mu.png

> 2. https://i.imgur.com/ftMlGq8.png

[0] https://news.ycombinator.com/item?id=35012981

Re: fMRI-to-image with contrastive learning and diffusion priors

#17
For context, early vision is easier to map than you might expect.

Here's a radiograph of the primary visual cortex created in 1982 by projecting a pattern onto a macaque's retina: https://web.archive.org/web/20100814085656im_/http://hubel.m...

An injection of radioactive sugar lets you see where the neurons were firing away and metabolizing the sugar.

(https://pubmed.ncbi.nlm.nih.gov/7134981/)

Re: fMRI-to-image with contrastive learning and diffusion priors

#18
post #6

Human communication will change dramatically once useful invasive brain-computer interfaces are available. People will suddenly realize that the reason language is primarily serial is simply due to the fact that it must be conveyed by a series of sounds. There will likely be a new type of visual language used via BCI "telepathy". It may have some ordering but will not rely so heavily on serializing information, since…

Indeed, it reminds me of the movie Arrival (and the short story upon which it's based) where the heptapods are able to show a complete sentence and story within one glyph. I thought it was interesting just how much the movie focused on linguistics, which is rare to see in Hollywood films.

Something else that's interesting about language is it's just a form of compressive medium for thoughts; I think of a concept, then I put it into words (compression) that you the listener then have to interpret and understand (decompression) and then fit your brain state to the new data you've received. It's overall a very lossy medium compared to what brains can do. It would be much easier to beam my thoughts and images and videos in my mind directly to you.

Unless you or I have aphantasia, of course.

Re: fMRI-to-image with contrastive learning and diffusion priors

#19

Earlier quoted context omitted.

I want to know the scifi You may base64 to spoiler proof it

I think it's "VGhlIFRocmVlLUJvZHkgUHJvYmxlbQ==" The aliens cannot lie to each other (they don't even have the idea of a lie), because their thoughts are transparent to each other.

Sounds like a recipe for conflict.
Post reply on HN