Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-res image reconstruction with latent diffusion models from human brain

github.com

Post reply on HN
Print view View original

101–110 of 170 posts

Re: High-res image reconstruction with latent diffusion models from human brain

#101

Post by pedrosorio »

groestl wrote: post #84
xmonkee wrote: post #42

Earlier quoted context omitted.

I think the point is that it's not a reconstruction. It's more like recognizing which letter of a thousand-letter alphabet is shown to the human after decoding their brain waves. Still impressive, but not really as impressive as visual reconstruction.

TBH, I was not impressed up until now, but given the videos I have in mind from people trying to use brain computing interfaces to type a text, now I'm impressed.

fMRI is useless for that purpose - latency is much higher than any BCI method you might’ve seen in those videos

Re: High-res image reconstruction with latent diffusion models from human brain

#102

Post by ricudis »

andai wrote: post #43
2bitencryption wrote: post #19

Are any of the example images novel, i.e. new to the model? Or is the model only reconstructing images it has already seen before? Either way, if I'm understanding right, it's very impressive. If the only input to the model (after training) is a fMRI reading, and from that it can reconstruct an image, at the very least that shows it can strongly correlate brain patterns back to the original image. It'd be even cooler…

There was a video many years ago (early 2010s?) demoing a similar technology, which would overlay and blend many images on top of each other to make a fuzzy image approximating what was actually being viewed. Edit: found it! https://youtu.be/nsjDnYxJ0bo

The youtube video quotes a paper by the same author, so it's probably the same group's work. I wonder why didn't they used an approach similar to the one in the video using SD - it looks more viable.

Re: High-res image reconstruction with latent diffusion models from human brain

#103

Post by qumpis »

darawk wrote: post #98
hiddencost wrote: post #67

Earlier quoted context omitted.

Nope. If you train a model where the input is an integer between 1 and 10, and the output is a specific image from a set of ten, the model will be able to get zero loss on the task. That is what's happening here.

Are you saying the demonstrated results are all in sample? Because this is definitely not true for out of sample data. And the GP comment implies that there is in fact a validation/holdout set.

I'm also confused by this. If everything was done properly, test results on the holdout set would've been shown. Wasnt that the case?

Re: High-res image reconstruction with latent diffusion models from human brain

#104

Post by flockonus »

notfed wrote: post #49
donohoe wrote: post #29

As people and groups increasingly move this direction do we think about vectors for abuse in 10, 20 or 50+ years? The human mind is considered the only place where we have true privacy. All these efforts are taking that away. At this rate all notions of privacy will soon be dead.

If this technology becomes accessible to courtrooms or police, they will use it. There will never be a way to encrypt thoughts.

With all the amazing advances we've seen in the recent years, I'd hope people would now stop thinking "there will never be X".

Re: High-res image reconstruction with latent diffusion models from human brain

#105

Post by Baeocystin »

squeaky-clean wrote: post #77
Filligree wrote: post #54

Earlier quoted context omitted.

People only think in words right before they say something, so I'm not sure how big a deal this is. I guess they'd be able to predict what I'm writing half a second before I write it? Would be useful if I lost the ability to write or speak, for whatever reason.

I'm "thinking in words" this entire thread as I read it. Do some people read without hearing the words in their head?

I don't have a word-based internal dialogue. When it's time to use words, like writing this post, sure, it's word time, so words are used.

But if I'm just reading, I just take in chunks and phrases while constructing a meaning model. The sounds themselves (or even individual words) don't really enter into it.

Re: High-res image reconstruction with latent diffusion models from human brain

#106

Post by brucethemoose2 »

Aransentin wrote: post #15

I immediately found the results suspect, and think I have found what is actually going on. The dataset it was trained on was 2770 images, minus 982 of those used for validation. I posit that the system did not actually read any pictures from the brains, but simply overfitted all the training images into the network itself. For example, if one looks at a picture of a teddy bear, you'd get an overfitted picture of anot…

Its still picking out the correct "overfitted" images, which is remarkable.

Theoretically, the results would scale to more training images... we just need to fMRI all of LAION-5B. Easy peasy.

Re: High-res image reconstruction with latent diffusion models from human brain

#107

Post by the_af »

nhchris wrote: post #22
zepolen wrote: post #13

Earlier quoted context omitted.

The only thing that's ethically questionable are humans themselves and such a device would likely do more to expose the unethical. Everybody remembers what happen when online dna hit mainstream.

I think only good thoughts so I'm not worried. Plus anyone with control of this technology is sure to have our best interests in mind.

Exactly.

Think good, happy thoughts. Happiness is mandatory. Being unhappy is treason. Treason is punishable by death.

Have a happy daycycle, citizen!

Re: High-res image reconstruction with latent diffusion models from human brain

#109

Post by angusturner »

Double_a_92 wrote: post #39
Hakkin wrote: post #28

Earlier quoted context omitted.

I'm definitely not an expert in this subject, but even if the model is overfitted, doesn't the fact that it can pull out the similar images at all give credit to the idea that a larger, non-overfitted model could actually work as the paper describes? It means that there does exist some correlation between the shown subject, the captured fMRI data, and the resulting location in latent space.

The output part is basically nonsense. It would be more honest if the output was a text. E.g. "Teddybear" instead of a bad image of a random teddybear.

Largely agree with this, although I think it would be interesting to formulate in terms of: "what is the mutual information between the fMRI scan and the stimulus".

i.e) is there actually more information than a few bits encoding a crude object category, which stable diffusion then hallucinates the rest (/ uses to regurgitate an over-fit image)?

Or are there many bits, corresponding spatially to different regions of the stimulus - allowing for some meaningful degree of generalization.

Re: High-res image reconstruction with latent diffusion models from human brain

#110

Post by geysersam »

hiddencost wrote: post #67
Hakkin wrote: post #28

Earlier quoted context omitted.

I'm definitely not an expert in this subject, but even if the model is overfitted, doesn't the fact that it can pull out the similar images at all give credit to the idea that a larger, non-overfitted model could actually work as the paper describes? It means that there does exist some correlation between the shown subject, the captured fMRI data, and the resulting location in latent space.

Nope. If you train a model where the input is an integer between 1 and 10, and the output is a specific image from a set of ten, the model will be able to get zero loss on the task. That is what's happening here.

Yes but the input isn't an integer from 1 to 10 right? It's MRI data.

Although it seems they're only able to extract the subject of the brain activity, not any actual "pictures".

Post reply on HN