A novel approach to neural machine translation
code.facebook.com
A novel approach to neural machine translation
1–10 of 45 posts
Re: A novel approach to neural machine translation
#2Re: A novel approach to neural machine translation
#3I really like that Facebook open sources both code and model along with the paper. Most companies don't: e.g. Google, deepmind, Baidu.
Re: A novel approach to neural machine translation
#4paper: https://s3.amazonaws.com/fairseq/papers/convolutional-sequen... code: https://github.com/facebookresearch/fairseq pre-trained models: https://github.com/facebookresearch/fairseq#evaluating-pre-t...
One logical continuation of adding more attention steps is to make decision of how many attention steps to take determined by the network ala "Adaptive Computation Time for Recurrent Neural Networks", are you planning to go in that direction?
Re: A novel approach to neural machine translation
#5> Facebook's mission of making the world more open
That's a rather strong statement, for a company that has become one of the world's most complained-about black boxes.
But yes, they have done a lot of good in the computer science space.
Re: A novel approach to neural machine translation
#6TLDR: Cutting edge accuracy, nine times faster than previous state of the art, published models and source code.
But go read the article- nice animated diagrams in there.
Re: A novel approach to neural machine translation
#7I'm relatively novice to machine learning but here's my best attempt to summarize what's going on in layman's terms. Please correct me if I'm wrong.
- Encode the words in the source (aka embedding, section 3.1)
- Feed every run of k words into a convolutional layer producing an output, repeat this process 6 layers deep (section 3.2).
- Decide on which input word is most important for the "current" output word (aka attention, section 3.3).
- The most important word is decoded into the target language (section 3.1 again).
You repeat this process with every word as the "current" word. The critical insight of using this mechanism over an RNN is that you can do this repetition in parallel because each "current" word does not depend on any of the previous ones.
Am I on the right track?
Re: A novel approach to neural machine translation
#8As far I understood it, Facebook put lots of research into optimizing a certain type of neural network (CNN), while everyone else is using another type called RNN. Up until now, CNN was faster but less accurate. However FB has progressed CNN to the point where it can compete in accuracy, particularly in speech recognition. And most importantly, they are releasing the source code and papers. Does that sound right?
Can anyone else give us an ELI5?
Re: A novel approach to neural machine translation
#9> Facebook's mission of making the world more open That's a rather strong statement, for a company that has become one of the world's most complained-about black boxes. But yes, they have done a lot of good in the computer science space.
> Facebook's mission of making the world more open
Like many big companies, they want to commoditize their products' complements.
"Smart companies try to commoditize their products' complements." https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/
Re: A novel approach to neural machine translation
#10TLDR: Cutting edge accuracy, nine times faster than previous state of the art, published models and source code. But go read the article- nice animated diagrams in there.
Haha, this really was TLDR-length. Kudos!