Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

fMRI-to-image with contrastive learning and diffusion priors

stability.ai

Post reply on HN
Print view View original

31–40 of 69 posts

Re: fMRI-to-image with contrastive learning and diffusion priors

#31

Post by dehrmann »

satvikpendem wrote: post #16

Wasn't there something similar a few months ago on HN and where the top comment talked about how it's not as impressive as it sounds [0]? The main issue is that this type of methodology is pulling from a pool of images, not literally reconstructing what image was seen in the brain directly. > I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2…

That one was a bit like not hotdogs: https://www.youtube.com/watch?v=ACmydtFDTGs

Re: fMRI-to-image with contrastive learning and diffusion priors

#33

Post by arketyp »

ilaksh wrote: post #27
satvikpendem wrote: post #18

Earlier quoted context omitted.

Indeed, it reminds me of the movie Arrival (and the short story upon which it's based) where the heptapods are able to show a complete sentence and story within one glyph. I thought it was interesting just how much the movie focused on linguistics, which is rare to see in Hollywood films. Something else that's interesting about language is it's just a form of compressive medium for thoughts; I think of a concept, the…

Right and I would go so far as to say that most types of intelligence are a type of functional compression also. There's definitely room for direct transfer of concrete unrolled information. But at the same time we would still need some forms of abstraction in many cases. I think the biggest issue with the compression of natural language is that the loss is different for each person, since everyone's "codec" varies.…

I wonder what the limits are to such a universal codec. From what I've gathered about synaesthesia (e.g. from V.S. Ramachandran, or Galton earlier), it varies quite significantly between persons. I believe it's said that some 3% of people have aphantasia for instance. That means entire modalities would be excluded for some in a latent space glyph language. Unless, I suppose, one could find ways of stimulating the synaesthetic connections artificially too.

Re: fMRI-to-image with contrastive learning and diffusion priors

#34

Post by tmabraham »

robg wrote: post #3

Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.

We think it could be useful for clinical research and maybe even diagnostics. For example, you could imagine a person with depression(or other neurological disorders) may have a different perception of the same image than a healthy person. Now with the much higher fidelity that both more powerful MRI machines and better generative AI tools can provide, this may now be a very promising direction for future research.

Re: fMRI-to-image with contrastive learning and diffusion priors

#35

Post by tmabraham »

theaiquestion wrote: post #15

I think the method of merging the pipelines via img2img should use controlnet. Possibly needing to be finetuned specifically for this, although existing controlnet models might work fine for this. This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.

Yes we've been looking into ControlNet as well, and I think there is one recent fMRI-to-image paper that also has tried ControlNet. Maybe we'll use ControlNet in MindEye v2 :)

Re: fMRI-to-image with contrastive learning and diffusion priors

#36

Post by wiz21c »

puchatek wrote: post #32

As someone suffering from intrusive thoughts I do not look forward to a future where other people can see what I sometimes see in my head.

As a person who is normal by any measuring standard, I do not look forward to a future where other people can see what I sometimes see in my head.

You'd be quite surprised.

Re: fMRI-to-image with contrastive learning and diffusion priors

#37

Post by atom_101 »

theaiquestion wrote: post #15

I think the method of merging the pipelines via img2img should use controlnet. Possibly needing to be finetuned specifically for this, although existing controlnet models might work fine for this. This is exactly what you'd want to use controlnet for - mapping semantic information onto the perceived structure.

Yes controlnet will be used in the next version. For this one we couldn't get it working in time.

Re: fMRI-to-image with contrastive learning and diffusion priors

#39

Post by atom_101 »

hospadar wrote: post #4

This is SO COOL. I'd guess (I did analysis for an fMRI lab for a year so I'm not a pro but not totally talking out of my orifice) that detecting images like this is among the easier things you could do (it probably wouldn't be so easy to do things like "guess the words I'm thinking of") and I suspect other sensory stuff might be harder but I have little knowledge there. One of the biggest issues with any attempt to e…

We did have a face reconstruction project planned. It is on the back-burner for now. That one will be based on something like the Celeb-A dataset instead of the Natural Scenes Dataset (images from MS-COCO) used here.

Re: fMRI-to-image with contrastive learning and diffusion priors

#40

Post by atom_101 »

RC_ITR wrote: post #29
jph00 wrote: post #24

Earlier quoted context omitted.

Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases…

> To achieve the goals of retrieval and reconstruction with a single model trained end-to-end, we adopt a novel approach of using two parallel submodules that are specialized for retrieval (using contrastive learning) and reconstruction (using a diffusion prior). What you can think of contrastive learning as is: two separate models that take different inputs and make vectors of the same length as outputs. This is ach…

This is mostly correct, except that there is only one model. This model takes an fMRI and predicts 2 outputs. The first is specialized for retrieval and the second can be fed into a diffusion model to reconstruct images.

You can see the comparison in performance between LAION-5B retrieval and actual reconstructions in the paper. When retrieving from a large enough database like LAION-5B, we can get images that are quite similar to the seen images in terms of high level content, but not so similar in low-level details (relative position of objects, colors, texture, etc). Reconstruction with diffusion models does much better in terms of low-level metrics.

Post reply on HN