Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine translation of cortical activity to text with encoder–decoder framework

nature.com

Post reply on HN
Print view View original

41–50 of 93 posts

Re: Machine translation of cortical activity to text with encoder–decoder framework

#41

Post by lioeters »

bognition wrote: post #10

Really cool to see progress made here but this won't be available for public use any time soon (likely decades). One of the biggest challenges with decoding brain signals is getting a large number sensors that detect voltages from a very localized region of the brain. This study was done with ECoG (Eletro-Cortico-Gram) which involves implanting small electrodes directly on the surface of the brain. Nearly all consume…

I've been fascinated by this topic since a couple decades ago, when I participated in a research study involving EEG and word recognition. Progress seems to be slow but steady, with applications getting more accurate and practical.

Your point about invasive implants being impractical for commercial use made me wonder.. I searched the first phrase that popped into my head, "Non-invasive Brain-Computer Interface". Looks like there's promising research on significantly improving sensitivity/resolution of EEG singals.

- First Ever Non-invasive Brain-Computer Interface Developed - https://www.technologynetworks.com/informatics/news/first-ev... (2019)

- Noninvasive neuroimaging enhances continuous neural tracking for robotic device control - https://robotics.sciencemag.org/content/4/31/eaaw6844

Still, your prediction of "likely decades" sounds realistic. I'm hoping for an affordable, non-invasive brain-computer interface to be as widely used as the keyboard, mouse, or microphone.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#42

Post by stanfordkid »

hughw wrote: post #34
bognition wrote: post #10

Really cool to see progress made here but this won't be available for public use any time soon (likely decades). One of the biggest challenges with decoding brain signals is getting a large number sensors that detect voltages from a very localized region of the brain. This study was done with ECoG (Eletro-Cortico-Gram) which involves implanting small electrodes directly on the surface of the brain. Nearly all consume…

Seismologist here. The problems you describe sound reminiscent of those in seismic imaging. Ideally you'd bury geophones below the weathered layer (typically 3 m) at the surface, to get ideal coupling. In practice that's not economic at scale, and so you plant them on the surface, and account for the non-linear wave transmission through the weathered layer by clever math and by collecting more samples. There's been a…

I looked into the problems in this field in the past, my impression is that the difficulty lies in how the signal diffuses through the skull and top-most layers of the brain. It’s all about resolution, and this sounds like a very similar problem. The question always is how much non-linearity is there, and what’s the frequency and density of the sensors relative to how often you have to sample to characterize it. Very excited to see how the field develops!

Re: Machine translation of cortical activity to text with encoder–decoder framework

#43

Post by jawns »

Ajedi32 wrote: post #37
bognition wrote: post #26

Earlier quoted context omitted.

I think they're a cool company and I even thought about applying. However the limiting factor here isn't the implant technology. Lots of groups have been building really great stuff for years (groups around the world have been recording from neurons in living animals for decades). The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. Right now the limit factor…

> However, the limiting factor here isn't the implant technology. [...] The issue is there are ethical implications for performing such an invasive brain surgery on a healthy person. That sounds like a limitation of the implant technology to me. The reason there are ethical problems with performing invasive brain surgery on a healthy person is because the risks and downsides of currently available implant technology…

> If getting a brain implant were as cheap, easy, and safe as getting a tattoo, the ethical problem would be largely solved.

But how realistic is that?

Your brain is a vital organ. It's encased in a hard skull. There is very little margin for error.

It just doesn't strike me as the sort of procedure that could ever be made as cheap, easy, and safe as getting a tattoo -- at least not in our lifetime.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#44

Post by hrgiger »

I have tried something similar 5 years ago using the meditation device from choosemuse.com. It was the cheapest option and provided hackable interface that you had access all the data. Then I wrote a small mobile app connects to headset.

Application was picking and showing a single random word from "hello world my name is hrgiger" then showing a greeen light, when I see green light, i think about the word and blink, headset was able to detect blinks as well so app was creating training data using blink time - xxx millis. So I created few thousands training data with 6 class using this and trained with my half-ass nn implementation and used generated weights to predict same way via mobile. Never achieved higher than 40%, tried all mixed waves, raw data, different windows of time series. Yet still it was a fun project to mess with, still I try to tune this nn implementation. If they achieve a practical solution I would use subtitles for the full length training. Simple netflix browser plugin might do the trick,but I am not sure if there will be a single AI algo that would understand everyones different data.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#46

Post by andai »

hrgiger wrote: post #44

I have tried something similar 5 years ago using the meditation device from choosemuse.com. It was the cheapest option and provided hackable interface that you had access all the data. Then I wrote a small mobile app connects to headset. Application was picking and showing a single random word from "hello world my name is hrgiger" then showing a greeen light, when I see green light, i think about the word and blink,…

The Muse headband looks like it covers a pretty small area of the head right? Other products in a similar price range cover more or less the whole scalp.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#47

Post by king07828 »

raverbashing wrote: post #35
king07828 wrote: post #32

Earlier quoted context omitted.

Has there been any work to use a neural network to generate/simulate ECoG signal output from an EEG signal input? (My Google-fu only gives definitions and distinctions for ECoG and EEG). Almost sounds similar in concept to deep learning super sampling (DLSS), i.e., taking a low resolution image/signal (EEG) and using a neural network to generate/simulate a high resolution image/signal (ECoG).

The problem sounds similar but it's completely different DLSS kinda works because we "know" what is what thing in a photo. EEG to ECOG would be trying to figure out a painting (that could be anything, from any painter) from a significant distance behind a frosted glass

> DLSS kinda works because we "know" what is what thing in a photo.

I thought the reason DLSS works is because the same rendering algorithm is used to generate the low resolution image and the high resolution image and the neural network merely learns a filter between the two.

Take a patient with ECoG implant(s), put EEG sensors on the patient, and hit record. You now have the same rendering mechanism (the brain) generating a low resolution signal (EEG) and a high resolution signal (ECoG).

However, back to DLSS, if the low resolution signal is a single pixel, then generating a 4k image from just that single pixel may not be very fruitful.

Still, it would be interesting to see an attempt at using a generative adversarial network (GAN) to generate an ECoG from an EEG. And if it doesn't work, then make a determination of how much more EEG sensitivity is needed before it will work.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#48

Post by erohead »

2008guy wrote: post #38
mattkrause wrote: post #25

Earlier quoted context omitted.

Is that true? There was a throwaway comment about that at a press conference and lots of rumors (positive and negative), but the white paper only mentions “monkey” once—-and in a reference to another group’s paper.

Elon musk said on stage that they had put it in a monkey and had the monkey move a cursor with it. I’ve been following musk very closely since 2010 and he’s never blatantly lied so I think this will not be the first time. He has made errors about timelines and etc but never, ever blatantly lied about something so concrete. And it’s plausible to boot.

> never blatantly lied.

Funding secured

Re: Machine translation of cortical activity to text with encoder–decoder framework

#49

Post by hrgiger »

andai wrote: post #46
hrgiger wrote: post #44

I have tried something similar 5 years ago using the meditation device from choosemuse.com. It was the cheapest option and provided hackable interface that you had access all the data. Then I wrote a small mobile app connects to headset. Application was picking and showing a single random word from "hello world my name is hrgiger" then showing a greeen light, when I see green light, i think about the word and blink,…

The Muse headband looks like it covers a pretty small area of the head right? Other products in a similar price range cover more or less the whole scalp.

Correct, if I recall correctly it has 8 sensors, 4 forehead and 4 ear side. Previous year I wanted to try again but seemed like they drop linux support and asked to reach them via mail to have sdk, so I didnt bother. If i would go now I would go with openbci project I guess

Re: Machine translation of cortical activity to text with encoder–decoder framework

#50

Post by linschn »

hrgiger wrote: post #44

I have tried something similar 5 years ago using the meditation device from choosemuse.com. It was the cheapest option and provided hackable interface that you had access all the data. Then I wrote a small mobile app connects to headset. Application was picking and showing a single random word from "hello world my name is hrgiger" then showing a greeen light, when I see green light, i think about the word and blink,…

40% over 6 classes is way above a random baseline. This is actually pretty cool. Congratulations!
Post reply on HN