Here's an old approach doing something similar, for reference: [1] [1] https://www.youtube.com/watch?v=nsjDnYxJ0bo
Without knowing much about (f)MRI scanning, I wonder how they get so much detail when the resolution seems so basic.
Real-Time Brain-to-Image Reconstructions
21–27 of 27 posts
Re: Real-Time Brain-to-Image Reconstructions
#22Not much to see there. I don't have much faith that the stated goal is realizable, I don't think fMRI data holds enough information for faithful reconstruction. What it will end up being, like all projects like it before, is a massive network that contains: - An implicit multi-class classifier that returns 1 out of n previously seen images - Gigantic model on top to hallucinate a reconstruction close to that previous…
I don’t know. Now that you explain how it works, it’s starting to worry me. Imagine being hooked up to an ML model designed to classify whether you’re thinking about porn or not, and emits a beep whenever it detects it. The implications of that are astonishing, if it’s accurate. It hinges on accuracy. If it can detect a topic, the image reconstruction is almost moot, because you can start interrogating people about w…
> it’s starting to worry me
Not any time soon: https://duckduckgo.com/?q=fMRI+device&ia=images&iax=images
Re: Real-Time Brain-to-Image Reconstructions
#23Not much to see there. I don't have much faith that the stated goal is realizable, I don't think fMRI data holds enough information for faithful reconstruction. What it will end up being, like all projects like it before, is a massive network that contains: - An implicit multi-class classifier that returns 1 out of n previously seen images - Gigantic model on top to hallucinate a reconstruction close to that previous…
There's already been preprints released showing fMRI reconstructions that appear to do better than an implicit multi-class classifier [1] [2]. But also, even if the result is an implicit multi-class classifier, if the n is sufficiently high then that would still be quite impressive!
Re: Real-Time Brain-to-Image Reconstructions
#24Not much to see there. I don't have much faith that the stated goal is realizable, I don't think fMRI data holds enough information for faithful reconstruction. What it will end up being, like all projects like it before, is a massive network that contains: - An implicit multi-class classifier that returns 1 out of n previously seen images - Gigantic model on top to hallucinate a reconstruction close to that previous…
There's already been preprints released showing fMRI reconstructions that appear to do better than an implicit multi-class classifier [1] [2]. But also, even if the result is an implicit multi-class classifier, if the n is sufficiently high then that would still be quite impressive! [1] https://openreview.net/pdf?id=pHdiaqgh_nf [2] https://arxiv.org/pdf/2211.06956.pdf
I see no evidence that they do better than multi-class classification, in fact they both work as I described. They learn embeddings of fMRI which perform an implicit classification of the data (which can be recovered just by quantizing the embedding space to get its modes) and put very large generation models on top.
The only reason the reconstructions are much better than before is because they use the latest generation models. Those models have internal models of the classes which allow them to fill in the high-frequency details in the reconstruction. The only information they get from the fMRI is the same low-frequency signal that previous papers already had, and indeed the only things the reconstructions get right are low-frequency: class of object/scene, broad position of object, broad shape of object.
fMRI scans are aggregates of brain information, they act like low-pass filters over the brain state. You can put as big a model on top as you want it won't make it more truthful a reconstruction.
I think, as you say, that detecting as many classes as possible is already a pretty good goal, developing new embeddings and techniques to see how much juice we can squeeze out of the scans. I like the arxiv preprint you posted in particular since it does just that and evaluates accuracy (although the way it does it is flawed since it uses an image classifier on the reconstruction which presents the same problems). What I don't like is the misrepresentation of what's going on when people put those large generative models on top of this kind of data.
Re: Real-Time Brain-to-Image Reconstructions
#25As someone with aphantasia I'd love to see this used to work out what it is that our brains aren't doing.
I can't even imagine (no pun intended) what would be like having aphantasia. It would be weird to me not being able to draw from memory a map of my own home. Or not being able to answer a question like "if you enter your home, walk straight ahead until you reach a wall, and then turn 90° counterclockwise, what would you be looking at?". Unless I'm wildly misunderstanding what "having a mental image" means.
Sorry for the delayed reply, but if you happen to come back... In my case, I can definitely do the two things you mention there (funnily enough most people with aphantasia actually have better than average spatial awareness), but there is no mental image of any of it - I can't see anything in my head, but have an entirely different "sense" of it all. The best description I've managed to come up with for my own attempts to mentally see something is that I can hold the essence of the object in my head. Ask me to imagine an apple and it's kinda like my brain has stored the underlying code that when run would render an apple, but the renderer is broken. I know what an apple is, I know what it looks like, I could describe one with great detail, but I can't picture one in my head.
Re: Real-Time Brain-to-Image Reconstructions
#26Earlier quoted context omitted.
I can't even imagine (no pun intended) what would be like having aphantasia. It would be weird to me not being able to draw from memory a map of my own home. Or not being able to answer a question like "if you enter your home, walk straight ahead until you reach a wall, and then turn 90° counterclockwise, what would you be looking at?". Unless I'm wildly misunderstanding what "having a mental image" means.
Sorry for the delayed reply, but if you happen to come back... In my case, I can definitely do the two things you mention there (funnily enough most people with aphantasia actually have better than average spatial awareness), but there is no mental image of any of it - I can't see anything in my head, but have an entirely different "sense" of it all. The best description I've managed to come up with for my own attemp…
Thanks for the response, that's more or less where I was leading with my message.
I don't think I have aphantasia myself, but what you say makes sense to me and describes my experience.
We might never know the answer, but for all we know, what you describe here might be what others call a "mental image", and maybe aphantasia is just a huge misunderstanding because the language we use to describe it makes it sound like it should be something more vivid that it really is.
Or maybe I also have aphantasia myself and I'm just finding out now.
These mind topics are always fascinating.
Re: Real-Time Brain-to-Image Reconstructions
#27Earlier quoted context omitted.
Sorry for the delayed reply, but if you happen to come back... In my case, I can definitely do the two things you mention there (funnily enough most people with aphantasia actually have better than average spatial awareness), but there is no mental image of any of it - I can't see anything in my head, but have an entirely different "sense" of it all. The best description I've managed to come up with for my own attemp…
Thanks for the response, that's more or less where I was leading with my message. I don't think I have aphantasia myself, but what you say makes sense to me and describes my experience. We might never know the answer, but for all we know, what you describe here might be what others call a "mental image", and maybe aphantasia is just a huge misunderstanding because the language we use to describe it makes it sound lik…
There's a test you can do here [0] which is quite interesting in showing the variation in people's internal experience. I did it with my wife and daughter who are both very visual thinkers - I was astonished at their answers and how different their mind's eyes appear to be to my own, they were similarly astonished at mine.