Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Machine translation of cortical activity to text with encoder–decoder framework

nature.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Machine translation of cortical activity to text with encoder–decoder framework

#61

Post by astrea »

carapace wrote: post #59

I'm pretty sure you can get that with a HD camera or two and some hypnosis plus off-the-shelf ML. One of the very first things I learned when I was studying hypnosis was to induce a simple binary signal from the unconscious. (Technically it's trinary: {y,n,mu} https://en.wikipedia.org/wiki/Mu_(negative) ) (In my case my right arm would twitch for "yes", left for "no", no twitch for "mu" (I don't want to go on a long…

Could you elaborate a little bit on the 'induce a simple binary signal from the unconscious'? That sounds fascinating.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#62

Post by hrgiger »

carapace wrote: post #55
hrgiger wrote: post #49

Earlier quoted context omitted.

Correct, if I recall correctly it has 8 sensors, 4 forehead and 4 ear side. Previous year I wanted to try again but seemed like they drop linux support and asked to reach them via mail to have sdk, so I didnt bother. If i would go now I would go with openbci project I guess

I've got a muse on my desk right now, it works-for-me on Ubuntu 16.04 with a BLE dongle and the BGAPIbackend of pygatt. FWIW, the main advantage of the muse design is that by going across the forehead it avoids interference from your hair, if I understand correctly. The NeuroTechX ppl are working to get some demo notebooks working with muse. https://github.com/NeuroTechX/eeg-notebooks https://neurotechx.com/

Thanks for the info, considering situation maybe I can give a shot again:) Yes I also remember reading that sensors had pretty much identical results with emotiv as well. This project also looks pretty cool , thanks for sharing.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#63

Post by mattkrause »

bognition wrote: post #27
akanet wrote: post #17

Earlier quoted context omitted.

Neuralink is even more speculative - they want to embed like a thousand tiny metal filaments deep into your brain.

This isn't new at all. Groups have been doing this for decades. Neuralink wants to mass produce the devices.

People have been recording brain activity for decades, but the scale is very different. Until about 10-15 years ago, most labs were using a handful of electrodes (my PhD work, for example, was done with 1-2 at a time). With cheaper amplifiers, it's become more common to use arrays with ~32-96 electrodes and they're often implanted chronically in the brain.

Neuropixels are fairly new and offer ~300 channels (selectable from ~1000). The neuralink thing would increase this another 10-fold.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#64

Post by morelisp »

erohead wrote: post #48
2008guy wrote: post #38

Earlier quoted context omitted.

Elon musk said on stage that they had put it in a monkey and had the monkey move a cursor with it. I’ve been following musk very closely since 2010 and he’s never blatantly lied so I think this will not be the first time. He has made errors about timelines and etc but never, ever blatantly lied about something so concrete. And it’s plausible to boot.

> never blatantly lied. Funding secured

This is merely a mistatement of fact. Lying is what the poors do!

Re: Machine translation of cortical activity to text with encoder–decoder framework

#65

Post by carapace »

astrea wrote: post #61
carapace wrote: post #59

I'm pretty sure you can get that with a HD camera or two and some hypnosis plus off-the-shelf ML. One of the very first things I learned when I was studying hypnosis was to induce a simple binary signal from the unconscious. (Technically it's trinary: {y,n,mu} https://en.wikipedia.org/wiki/Mu_(negative) ) (In my case my right arm would twitch for "yes", left for "no", no twitch for "mu" (I don't want to go on a long…

Could you elaborate a little bit on the 'induce a simple binary signal from the unconscious'? That sounds fascinating.

Sure. (Thanks for asking.) It was one of the first things I learned when I started studying (self-)hypnosis. It's a simple way to have access to the unconsciousness without going into a trance.

There's really nothing to it. You induce a light trance and ask the unconscious mind to create a simple unambiguous yes-no signal. Finger motions are common. After that you can ask yourself questions and get y/n answers (or non-response, what I'm calling "mu", which indicates some issue with the phrasing or nature of the query.)

I should mention that you should be very careful about your self-model if you are experimenting with piercing the barrier between the conscious and unconscious minds. In computer terms, this signal corresponds to a kind of trans-mechanical oracle and having it available to your (metaphorical) Turing machine mind makes you into a fundamentally different kind of processor, operating by rules that may be unfamiliar.

https://en.wikipedia.org/wiki/Oracle_machine

But see also: https://en.wikipedia.org/wiki/Oracle because that's more accurate.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#66

Post by garethsprice »

antupis wrote: post #15
bognition wrote: post #10

Really cool to see progress made here but this won't be available for public use any time soon (likely decades). One of the biggest challenges with decoding brain signals is getting a large number sensors that detect voltages from a very localized region of the brain. This study was done with ECoG (Eletro-Cortico-Gram) which involves implanting small electrodes directly on the surface of the brain. Nearly all consume…

Stupid question but could you do something very simple like on/off-switch with EEG and something similar than what they have done here?

Muse headset uses the "on/off" signal for meditation training. Has a fun stream API you can play with.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#67

Post by carapace »

hrgiger wrote: post #62
carapace wrote: post #55

Earlier quoted context omitted.

I've got a muse on my desk right now, it works-for-me on Ubuntu 16.04 with a BLE dongle and the BGAPIbackend of pygatt. FWIW, the main advantage of the muse design is that by going across the forehead it avoids interference from your hair, if I understand correctly. The NeuroTechX ppl are working to get some demo notebooks working with muse. https://github.com/NeuroTechX/eeg-notebooks https://neurotechx.com/

Thanks for the info, considering situation maybe I can give a shot again:) Yes I also remember reading that sensors had pretty much identical results with emotiv as well. This project also looks pretty cool , thanks for sharing.

Cheers!

Re: Machine translation of cortical activity to text with encoder–decoder framework

#68

Post by posterboy »

bognition wrote: post #28
antupis wrote: post #15

Earlier quoted context omitted.

Stupid question but could you do something very simple like on/off-switch with EEG and something similar than what they have done here?

Not sure how that would change anything. Think of the brain as a crowd at a soccer field and to decode the brain you'd need a mic on each person to capture every conversation. ECoG is like having a mic shared between every 100 people. While EEG is like having a single mic that is 10K feet over the soccer stadium.

This is my new favorite metaphor, if it's an analogy to how people sing, and how a person can sing somewhat randomly (free) yet controlled. The fact that there are two sides to the party chanting against each other, and the stadion speaker blurting over it occasionally, makes it so much better.

Corallary, you need two microphones at least for simplicities sake, although the repetitivenes of the chants makes it a little easier.

Most likely this will have good use in aphasia research.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#69

Post by salawat »

stainforth wrote: post #57
2008guy wrote: post #40

Earlier quoted context omitted.

The inflammation and damage is seen in traditional arrays that are large and rigid. A thread with low enough moment of inertia will likely not cause as much damage. And the damage is only important if you put the electrodes in an important place... we currently screw two giant lag screws into people’s heads and call it “deep brain stimulation” so I feel optimistic about the long game. But you’d still be right to be w…

Can you continue about your opposition to it existing?

It's a social crisis waiting to happen, especially if you can end up decoding more than just what someone uses their speech centers to articulate.

This is invasive to the extreme, and seems to open the door for violations of people's intimate thoughts down the road.

You may not think about it much now, but if you pay any attention to things like intrusive thoughts, or even have to deal with carefully maintaining a public face in the workplace, it should not be difficult to realize why these technologies are legitimately dangerous even as read only systems.

The real nightmare begins when you finally get fed up with Read-Only and figure out how to write in order to potentially mutate mental state.

I'm normally pretty forward-thinking in terms of embracing the March of technological progress. However, the last decade or so has shown we as a society have had our grasp exceed our socio/ethical/moral framework for using it responsibly; and the potential abuse a full read/write neural interface would enable is one of the few things that has managed to attain a "full-stop" in my personal socio-ethical-moral framework.

Not to sound like that an adult, but we're just not ready.

Before anyone points out that the same moral outrage probably occurred with the printing press; there is a big damn difference between changing someone's mind through pamphlets, and having a direct link to the limbic system to tickle on a whim. We do a very bad job of correctly estimating the long-term effects of technological advancement; just look at how destructive targeted advertising has been.

I haven't reached my conclusion on an existing preconception/predisposition either. I used to be massively for this particular advancement. Only through a long time spent reflecting on it has my viewpoint done a 180.

I'm aware of all of the positive applications for the handicap, brain-locked, and paralyzed; but I'm still reluctant to consider embracing it for their sake when I've seen how prone to taking a crowbar to a minor exception/precedent our legal system is.

Maybe I've just been in the industry long enough not to trust tech people to keep society's overall well-being and stability at heart. Maybe I'm becoming a luddic coward as I get older. I don't know, and I ask myself if I'm not being unreasonable every day. The answer hasn't changed though in a long while, even though I do keep trying to seek out opportunities to challenge it.

I hope that helps, and doesn't make me sound like too much of a nut.

Re: Machine translation of cortical activity to text with encoder–decoder framework

#70

Post by lars »

kasmura wrote: post #22
lars wrote: post #9

This is cool. For those who are not super familiar with language processing, I think it's good to point out the limitations of what's been done here though. They mention that professional speech transcription has word error rate around 5%, and that their method gets a WER of 3%. Sure, but the big distinction is that speech transcription must operate on an infinite number of sentences, even sentences that have never b…

Is the following quote at odds with what you are saying about 50-way classification? "On the other hand, the network is not merely classifying sentences, since performance is improved by augmenting the training set even with sentences not contained in the testing set (Fig. 3a,b). This result is critical: it implies that the network has learned to identify words, not just sentences, from ECoG data, and therefore that…

The difficulty of the problem is that of a 50-way classification. If the only goal was to minimize WER, a simple post-processing step choosing the nearest sentence in the training set could easily bring the WER down further. They've chosen to do it the way they did it presumably to show that it can be done that way, and I don't fault them for it.

They claim that word-by-word decoding implies that the network has learned to identify words. This may well be true, but it isn't possible to claim that from their result. For example, let's say you average all electrode samples over the relevant timespan, transform that representation with a FFW neural net, and feed that into the an RNN decoder. It would still predict word-by-word, on a representation that necessarily does not distinguish between words (because the time dimension has been averaged over). Such a model can still output words in the right order, just from the statistics of the training sentences being baked into the decoder RNN.

Post reply on HN