A novel approach to neural machine translation • Hacker News Board

Re: A novel approach to neural machine translation

#41

Post by dooxoo » Wed, May 10, 2017, 3:07 PM UTC

jgehring wrote: post #36

deepnotderp wrote: post #15

As far as I understand, only the use of the attention mechanism with ConvNets is novel, right? Convolutional encoders have been done before.

Yes, there have been a couple of attempts to use CNNs for translation already, but none of them outperformed big and well-tuned LSTM systems. We propose an architecture that is fast to run, easy to optimize and can scale to big networks, and could thus be used as a base architecture for future research. There are a couple of contributions in the paper ( https://arxiv.org/abs/1705.03122 ) apart from demonstrating the…

> there have been a couple of attempts to use CNNs for translation already, but none of them outperformed big and well-tuned LSTM systems

It is true that QRNN had results on mostly small-scale benchmarks, but it seemed that Bytenet especially the second version had SOTA results both for language models with characters and for machine translation with characters on the same large-scale En-De WMT task that is used in this paper.

MT with characters, with regards to ordering, structure, etc, is potentially much harder than with words or word-pieces, since the encoded sequences are 5 or 6 times longer on average, and the meanings of words need to be built up from individual characters.

Re: A novel approach to neural machine translation

#42

Post by jgehring » Wed, May 10, 2017, 4:40 PM UTC

dooxoo wrote: post #41

jgehring wrote: post #36

Earlier quoted context omitted.

Yes, there have been a couple of attempts to use CNNs for translation already, but none of them outperformed big and well-tuned LSTM systems. We propose an architecture that is fast to run, easy to optimize and can scale to big networks, and could thus be used as a base architecture for future research. There are a couple of contributions in the paper ( https://arxiv.org/abs/1705.03122 ) apart from demonstrating the…

> there have been a couple of attempts to use CNNs for translation already, but none of them outperformed big and well-tuned LSTM systems It is true that QRNN had results on mostly small-scale benchmarks, but it seemed that Bytenet especially the second version had SOTA results both for language models with characters and for machine translation with characters on the same large-scale En-De WMT task that is used in t…

Yes, ByteNet v2 outperforms LSTMs on characters but not on word pieces. It would be interesting to see how our model performs on characters, especially when scaled up to the size of ByteNet (30+30 layers) and also how ByteNet performs on BPE codes. I think that character-level NMT is definitely interesting and worth investigating, but from a practical point of view it makes sense to choose a representation that achieves the maximum translation accuracy and speed.

Re: A novel approach to neural machine translation

#43

Post by beagle3 » Wed, May 10, 2017, 6:54 PM UTC

blacksmythe wrote: post #9

gavinpc wrote: post #5

> Facebook's mission of making the world more open That's a rather strong statement, for a company that has become one of the world's most complained-about black boxes. But yes, they have done a lot of good in the computer science space.

> Facebook's mission of making the world more open Like many big companies, they want to commoditize their products' complements. "Smart companies try to commoditize their products' complements." https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

And they innovated in this space with the "facebook patent grant" -- they give away free stuff, with a patent grant that disappears as soon as you sue them.

And they're better at marketing than many - heard of the amazing new zlib replacement, Zstd? It's better in every way except one - unlike zlib (unconditionally patent free), it is only patent free as long as you don't sue Facebook. But almost no one is aware of that.

Re: A novel approach to neural machine translation

#44

Post by denzil_correa » Wed, May 10, 2017, 11:37 PM UTC

sprobertson wrote: post #40

albertzeyer wrote: post #24

Earlier quoted context omitted.

Google has released some framework for translation: https://github.com/google/seq2seq/ DeepMind has also released some framework which also has all the building blocks for translation: https://github.com/deepmind/sonnet

Sure but these don't address parent's statement: they don't release code with research. These both came years after the original seq2seq paper.

Google's Seq2Seq came 3-4 months after the NMT paper.

Re: A novel approach to neural machine translation

#45

Post by etruong42 » Thu, May 11, 2017, 1:55 PM UTC

blacksmythe wrote: post #9

gavinpc wrote: post #5

> Facebook's mission of making the world more open That's a rather strong statement, for a company that has become one of the world's most complained-about black boxes. But yes, they have done a lot of good in the computer science space.

> Facebook's mission of making the world more open Like many big companies, they want to commoditize their products' complements. "Smart companies try to commoditize their products' complements." https://www.joelonsoftware.com/2002/06/12/strategy-letter-v/

Where is the complementary relationship? Is it with the facebook.com website and their open-source AI product? More AI means more people will use Facebook.com?

Or is the comment only tangential to OP?