Transformer: A Novel Neural Network Architecture for Language Understanding
research.googleblog.com
Transformer: A Novel Neural Network Architecture for Language Understanding
1–10 of 20 posts
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#2I'm a novice when it comes to neural network models, but would I be correct in interpreting this as a convolutional network architecture with multiple stacked encoders and decoders?
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#3I'm a novice when it comes to neural network models, but would I be correct in interpreting this as a convolutional network architecture with multiple stacked encoders and decoders?
Nah, the paper explicitly states that their system is not recurrent nor convolutional:
> To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using RNNs or convolution.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#4DeepL (was on HN earlier this week) also uses an attention-based mechanism like this (or at least, with the same intention and effect). They didn't really talk about it but the founder mentioned it to me. The two seem to have independently pursued the technique, perhaps from some shared ancestor like a paper they both were inspired by.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#5DeepL (was on HN earlier this week) also uses an attention-based mechanism like this (or at least, with the same intention and effect). They didn't really talk about it but the founder mentioned it to me. The two seem to have independently pursued the technique, perhaps from some shared ancestor like a paper they both were inspired by.
Attention is used in machine translation since 2014 as far as i know (Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio:
Neural Machine Translation by Jointly Learning to Align and Translate) in various forms.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#6The key to this paper is the "Multi-Head Attention" which looks a lot like a Convolutional layer to me.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#7does it mean we don't need gradient descent after all to achieve the same result?
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#8DeepL (was on HN earlier this week) also uses an attention-based mechanism like this (or at least, with the same intention and effect). They didn't really talk about it but the founder mentioned it to me. The two seem to have independently pursued the technique, perhaps from some shared ancestor like a paper they both were inspired by.
The paper linked in the blog was actually published back in June. And as the sibling comment says, people have been trying to use attention mechanisms to augment recurrent networks for some time. The novel idea here seems to be sorting the recurrent network, and just keeping the attention mechanism.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#9would something like this work well on mixed/pidgin languages - e.g. Hinglish , which is a mixture of Hindi and English and used in daily vernacular ?
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#10DeepL (was on HN earlier this week) also uses an attention-based mechanism like this (or at least, with the same intention and effect). They didn't really talk about it but the founder mentioned it to me. The two seem to have independently pursued the technique, perhaps from some shared ancestor like a paper they both were inspired by.
Discussion about DeepL: https://news.ycombinator.com/item?id=15122764
Attention is not new. Everyone uses it (for translation and many related tasks). It's very much the standard right now.
Avoiding recurrent connections inside the encoder or decoder is also not completely new. That came up when people tried to only use convolutions.
Googles Transformer was made public in June 2017, in the paper Attention is all you need, https://arxiv.org/abs/1706.03762, including TensorFlow code, https://github.com/tensorflow/tensor2tensor . Note that the new thing here is that they neither use recurrence nor convolution but rely entirely on self-attention instead, with simple fully-connected layers, in both the encoder and the decoder.
DeepL directly compares their model to Transformer, in terms of performance (BLEU score), here: https://www.deepl.com/press.html