Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

High-res image reconstruction with latent diffusion models from human brain

github.com

Post reply on HN
Print view View original

91–100 of 170 posts

Re: High-res image reconstruction with latent diffusion models from human brain

#91

Post by ditchfieldcaleb »

Hakkin wrote: post #81
ditchfieldcaleb wrote: post #74

Earlier quoted context omitted.

What are you talking about? They didn't train a model for this. That's why it's so impressive.

Quoting from the paper, The only training required in our method is to con- struct linear models that map fMRI signals to each LDM component, and no training or fine-tuning of deep-learning models is needed. ... To construct models from fMRI to the components of LDM, we used L2-regularized linear regression, and all models were built on a per subject basis. Weights were estimated from training data, and regularizatio…

Ah. This makes more sense. Thanks.

Re: High-res image reconstruction with latent diffusion models from human brain

#92

Post by crispyambulance »

In 1990, there was a train-wreck Wim Wenders movie that I loved and still love called "Until the End of the World". It was about a scientist (played by Max Von Sydow) who created a machine that could record someone's dreams or visual experiences directly from the brain and play them back even to a blind person. https://youtu.be/gilzgbdk300?t=442

Anyways, the images that were depicted in this work of fiction shot in 1990 about "the future" of 2000, had a very interesting look to them-- kind of distorted and dreamy like the images in the paper.

Are the images in the paper just a case of overfitting? ¯\_(ツ)_/¯ but it still makes me giddy remembering the Wim Wenders film.

Re: High-res image reconstruction with latent diffusion models from human brain

#93

Post by nottathrowaway3 »

judge2020 wrote: post #50
andai wrote: post #38

Earlier quoted context omitted.

There was a guy at MIT about ten years ago (edit: 2018! Woah) who made a headset that would read electrical impulses from your face. Apparently when people think in words, the same nerves fire as when they speak, just at a lower activation level. Using those signals it is possible to reconstruct the words being thought. I'm surprised it didn't seem to go anywhere. Edit: found it https://youtu.be/RuUSc53Xpeg

> I'm surprised it didn't seem to go anywhere. At least not publicly.

[dead]

Re: High-res image reconstruction with latent diffusion models from human brain

#95

Post by ryukafalz »

squeaky-clean wrote: post #77
Filligree wrote: post #54

Earlier quoted context omitted.

People only think in words right before they say something, so I'm not sure how big a deal this is. I guess they'd be able to predict what I'm writing half a second before I write it? Would be useful if I lost the ability to write or speak, for whatever reason.

I'm "thinking in words" this entire thread as I read it. Do some people read without hearing the words in their head?

This tends to start long threads with tons of anecdotes whenever it comes up but the answer is yes, some people do. (I usually do but not always, for example.)

For lots and lots of discussion, you can search for things like this on e.g. Reddit: https://www.reddit.com/r/NoStupidQuestions/comments/nosdwt/d...

Re: High-res image reconstruction with latent diffusion models from human brain

#96

Post by [deleted] »

moron4hire wrote: post #58
Hakkin wrote: post #46

Earlier quoted context omitted.

In this specific case I agree, since the model may be overfitted, it seems like it's currently just a glorified object classifier based on what was in the training data, but the fact that it works at all may indicate that the underlying idea has merit. They would probably have to train a much larger network to see if it's able to separate features distinctly enough using the input fMRI data to be useful.

It's not an object classifier at all. They had to text-prompt the system, first. I think the general idea is using the fMRI data as the pseudorandom initialization for the latent diffusion model to explore. From what I understand, regular Stable Diffusion starts by generating a noise and then hallucinating modifications of that noise to make less noise. The more you let it run, the better the results. So instead of j…

[deleted]

Re: High-res image reconstruction with latent diffusion models from human brain

#97

Post by antegamisou »

ryanjshaw wrote: post #83
antegamisou wrote: post #44

Earlier quoted context omitted.

> As people and groups increasingly move this direction do we think about vectors for abuse in 10, 20 or 50+ years? No, the delusional shortsighted and revenue-driven SV startup culture doesn't give a shit about such 'technophobic trivialities'.

Yu Takagi, Shinji Nishimoto Graduate School of Frontier Biosciences, Osaka University, Japan

Yeah, put the blame on the researchers and not the greedy worms VCs are.

Re: High-res image reconstruction with latent diffusion models from human brain

#98

Post by darawk »

hiddencost wrote: post #67
Hakkin wrote: post #28

Earlier quoted context omitted.

I'm definitely not an expert in this subject, but even if the model is overfitted, doesn't the fact that it can pull out the similar images at all give credit to the idea that a larger, non-overfitted model could actually work as the paper describes? It means that there does exist some correlation between the shown subject, the captured fMRI data, and the resulting location in latent space.

Nope. If you train a model where the input is an integer between 1 and 10, and the output is a specific image from a set of ten, the model will be able to get zero loss on the task. That is what's happening here.

Are you saying the demonstrated results are all in sample? Because this is definitely not true for out of sample data. And the GP comment implies that there is in fact a validation/holdout set.

Re: High-res image reconstruction with latent diffusion models from human brain

#99

Post by gptgpp »

not-my-account wrote: post #7
rvnx wrote: post #3

Creepy and cool at the same time. It goes into the bucket of things that are not ethically right, same ways as implanting chips to read monkeys brains. But technically interesting and well-executed.

Why is it unethical to put chips in monkey brains?

Oooh I love moral philosophy.

Here's one deontological perspective you could take:

It is always wrong to cause unecessary suffering to others.

Now, the subjective traits to be considered here are "unecessary" and "suffering."

It used to be a common belief that animals lacked the capacity to suffer as humans do. They could feel pain, nocioception sure, but whether it caused complex psychological suffering (torment) used to be contenious.

Today, this is certainly not contentious for our closest relatives. All primates possess a theory of mind, long memory, emotional states, complex social behavior like lasting bonds and altruism, and other traits necessary to suffer in significant ways.

So the focus (as long as we are considering primates, obviously nobody cares about model species like drosophilia as they have a greatly dimished capacity to suffer (edit: although I should mention that ranking things based on capacity to suffer leads to pretty awful territories too. Eg, if a human is severely mentally disabled, is more permissible to experiment on them? I think most of us would say no, which raises the question why it's okay to do so for other species)) shifts to whether causing them immense physical pain and torture is necessary.

And this is where I think things get pretty murky, and I will leave the rest up to you! I wish more people were curious about moral philosophy and creating their own consistent ethical framework for the world... I think it's especially important in science and engineering.

Of course, you could use a different model like utilitarianism, but utilitarianism still requires some level of deontological principles or you end up with a pretty extremist moral philosophy (same goes for just having Kantian deontology with no room for utilitarianism, IMO).

edit 2: come to think of it, Jainists would certainly have an issue with experimentation even on drosophilia, so I take that back that "nobody" cares. IIRC they even have a mouth covering to prevent swallowing and killing any insects that might accidentally fly into their mouths, as well as a specialized stick to gently move things like spiders and other insects out of the way. I know most people would scoff at that, but I think their deep respect for all forms of life is beautiful.

Re: High-res image reconstruction with latent diffusion models from human brain

#100

Post by gptgpp »

Tomis02 wrote: post #78
elil17 wrote: post #41

Earlier quoted context omitted.

Sound's like you're opposed to all animal research, not specifically brain-computer interface research. We also don't ask a monkey's consent before doing any other sort of experiment on it.

I'm not necessarily opposed to it, I simply answered OPs question. It's unethical because there's no consent being given. However, humans do animal research regardless because they view it as "the end justifies the means", and they usually try to use humane methods (which is also not really an excuse because you can't ask the animal if it feels pain, or depression, or anything else).

To be fair, technically it becomes ethical when your position is that "the end justifies the means."

IMO Utilitarianism is a particularly dangerous ethical framework when wielded by narcissists who have a difficult time imagining that they might catastrophically fail (looking at you Sam Bankman-Fried), or might not be delivering salvation to the world (looking at you Elon, lol).

If you tell yourself it's for "the good of humanity", or the alternative is destruction / widespread death, you can justify any action.

Post reply on HN