21–30 of 69 posts
Awesome, predicting words from fMRI has been around for a while and visual cortex can be mapped well. That said, and coming from a background in neuroimaging 20 years ago, what’s the applicability? MRI hasn’t gotten that much more cost effective for more widespread uses. Magnets are expensive.
Wasn't there something similar a few months ago on HN and where the top comment talked about how it's not as impressive as it sounds [0]? The main issue is that this type of methodology is pulling from a pool of images, not literally reconstructing what image was seen in the brain directly. > I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2…
As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases like LAION-5B. We demonstrate through ablations that MindEye's performance improvements over previous methods result from specialized submodules for retrieval and reconstruction, improved training techniques, and training models with orders of magnitude more parameters."
Note that LAION-5B has five billion images.
This is mind reading, we are in the future.
Wasn't there something similar a few months ago on HN and where the top comment talked about how it's not as impressive as it sounds [0]? The main issue is that this type of methodology is pulling from a pool of images, not literally reconstructing what image was seen in the brain directly. > I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2…
Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases…
Human communication will change dramatically once useful invasive brain-computer interfaces are available. People will suddenly realize that the reason language is primarily serial is simply due to the fact that it must be conveyed by a series of sounds. There will likely be a new type of visual language used via BCI "telepathy". It may have some ordering but will not rely so heavily on serializing information, since…
Indeed, it reminds me of the movie Arrival (and the short story upon which it's based) where the heptapods are able to show a complete sentence and story within one glyph. I thought it was interesting just how much the movie focused on linguistics, which is rare to see in Hollywood films. Something else that's interesting about language is it's just a form of compressive medium for thoughts; I think of a concept, the…
There's definitely room for direct transfer of concrete unrolled information. But at the same time we would still need some forms of abstraction in many cases.
I think the biggest issue with the compression of natural language is that the loss is different for each person, since everyone's "codec" varies. In other words, people often interpret language in different ways.
But suppose that humans or AIs or AI-enhanced humans could have exactly the same base dictionary or interpretive network or "codec" or whatever for a (visual or word-based) language. Then we could get away from many of the disputes and misunderstandings that arise purely from different interpretations.
Earlier quoted context omitted.
Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases…
If it's still retrieving an image and not reconstructing it, if the dataset is large enough that's decently fine, but this is generally not how diffusion models work in general and I'd have expected the model to map the fMRI data to a wholly new image.
They explain that they've done both retrieval and reconstruction, and have lots of pictures showing examples of each.
Wasn't there something similar a few months ago on HN and where the top comment talked about how it's not as impressive as it sounds [0]? The main issue is that this type of methodology is pulling from a pool of images, not literally reconstructing what image was seen in the brain directly. > I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2…
Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases…
What you can think of contrastive learning as is: two separate models that take different inputs and make vectors of the same length as outputs. This is achieved by training both models on pairs of training data (in this case fMRI images and observed images).
What the LAION-5B work shows is that they did a good enough job of this training that the models are really good at creating similar vectors for nearly any image and fMRI pair.
Then, they make a prior model which basically says “our fMRI vectors are essentially image vectors with an arbitrary amount of randomness in them (representing the difference between the contrastive learning models). Let’s train a model to learn to remove that randomness, then we have image vectors.”
So yes, this is an impressive result at first glance and not some overfitting trick.
It’s also sort of bread and butter at this point (replace fMRI with “text” and that’s just what Stable Diffusion is).
They’ll be lots of these sort of results coming out soon.
Earlier quoted context omitted.
Yes there was. However this is a different paper, describing a different method, applied to a different dataset, with different results. As the abstract says, "In particular, MindEye can retrieve the exact original image even among highly similar candidates indicating that its brain embeddings retain fine-grained image-specific information. This allows us to accurately retrieve images even from large-scale databases…
If it's still retrieving an image and not reconstructing it, if the dataset is large enough that's decently fine, but this is generally not how diffusion models work in general and I'd have expected the model to map the fMRI data to a wholly new image.