Machine translation of cortical activity to text with encoder–decoder framework
1–10 of 93 posts
Re: Machine translation of cortical activity to text with encoder–decoder framework
#2Can we remove the tracking query-string from this link, please? It works fine without it:
https://www.nature.com/articles/s41593-020-0608-8.epdf
Edit. Sorry, seems to only show the first page if you remove the token.
Re: Machine translation of cortical activity to text with encoder–decoder framework
#3Can we remove the tracking query-string from this link, please? It works fine without it: https://www.nature.com/articles/s41593-020-0608-8.epdf Edit. Sorry, seems to only show the first page if you remove the token.
Good idea.
I wonder if the site could run a URL cleaner on all link submissions?
@dang?
Re: Machine translation of cortical activity to text with encoder–decoder framework
#4[deleted]
Re: Machine translation of cortical activity to text with encoder–decoder framework
#5Looks like a good approach and the error rate of 3% is really good, I guess.
Did they mention how they got the input data? I couldn't find it.
Re: Machine translation of cortical activity to text with encoder–decoder framework
#6Fascinating work but far from what some might hope from reading only the title.
The translation is restricted to a vocabulary of 30 to 50 unique sentences.
Re: Machine translation of cortical activity to text with encoder–decoder framework
#7Looks like a good approach and the error rate of 3% is really good, I guess. Did they mention how they got the input data? I couldn't find it.
They use 250 ECG electrodes as input. I think that means it's above the skin, so not invasive.
Re: Machine translation of cortical activity to text with encoder–decoder framework
#8Fascinating work but far from what some might hope from reading only the title. The translation is restricted to a vocabulary of 30 to 50 unique sentences.
They do mention that the network is partially learning the words themselves:
> "On the other hand, the network is not merely classifying sentences, since performance is improved by augmenting the training set even with sentences not contained in the testing set (Fig. 3a,b). This result is critical: it implies that the network has learned to identify words, not just sentences, from ECoG data, and therefore that generalization to decoding of novel sentences is possible."
Re: Machine translation of cortical activity to text with encoder–decoder framework
#9This is cool. For those who are not super familiar with language processing, I think it's good to point out the limitations of what's been done here though. They mention that professional speech transcription has word error rate around 5%, and that their method gets a WER of 3%. Sure, but the big distinction is that speech transcription must operate on an infinite number of sentences, even sentences that have never been said before. This method only has to distinguish between 30-50 sentences, and the same sentences must exist at least twice in the training set and once in the test set. Decoding word-by-word is really a roundabout way of doing a 50-way classification here.
It's an invasive technique, so they need electrodes on a human cortex. This means data collection is costly, so their operating in very low data regime compared to most other seq2seq applications. It seems theoretically possible that this could operate on Google translate level accuracy if the sentence dataset was terrabyte sized rather than kilobyte sized. That dataset size seems very unlikely to be collected any time soon, so we'll need massive leaps in data efficiency in machine learning for something like this to reach that level. They explore transfer learning for this, which is nice to see. Subject-independent modelling is almost certainly a requirement to achieve significant leaps in accuracy for methods like this.
Re: Machine translation of cortical activity to text with encoder–decoder framework
#10Really cool to see progress made here but this won't be available for public use any time soon (likely decades).
One of the biggest challenges with decoding brain signals is getting a large number sensors that detect voltages from a very localized region of the brain. This study was done with ECoG (Eletro-Cortico-Gram) which involves implanting small electrodes directly on the surface of the brain. Nearly all consumer devices use EEG (Electro Encephelo Gram) which involves putting sensors on the surface of the skin.
Commercially available ECoG is highly unlikely as it requires an extremely invasive brain surgery. For ethical reasons the implants from the study we're likely implanted to help diagnose existing life threatening medical issues.
Decoding speech from EEG won't work as well as ECoG a number of reasons. First the physical distance between the sensors and the brain Means the signals you pick up aren’t localized. Second the skin and skull are great low pass filters and filter out really interesting signals at higher frequencies, 100-2KHZ. Additionally these signals have a really low signal power because they're correlated with neuronal spiking.
ECoG does a really good job picking on these signals because the sensor is literally on the surface of the brain. Its really hard to pick up these signals reliably with EEG.