Natural Language Processing: The Age of Transformers
blog.scaleway.com
Natural Language Processing: The Age of Transformers
1–10 of 26 posts
Re: Natural Language Processing: The Age of Transformers
#2Previous discussion on “Attention is all you need”: https://news.ycombinator.com/item?id=15938082
Recent work by Jakob and team:https://ai.google/research/people/author37567/
Image transformer is particularly interesting
Re: Natural Language Processing: The Age of Transformers
#3Great blog!
Re: Natural Language Processing: The Age of Transformers
#4Previous discussion on “Attention is all you need”: https://news.ycombinator.com/item?id=15938082 Recent work by Jakob and team: https://ai.google/research/people/author37567/ Image transformer is particularly interesting
Also, recent discussion about a very good intro article about Transformers: https://news.ycombinator.com/item?id=20773992
Re: Natural Language Processing: The Age of Transformers
#5Previous discussion on “Attention is all you need”: https://news.ycombinator.com/item?id=15938082 Recent work by Jakob and team: https://ai.google/research/people/author37567/ Image transformer is particularly interesting
Also, recent discussion about a very good intro article about Transformers: https://news.ycombinator.com/item?id=20773992
I was about to share it too, really GREAT intro :)
Re: Natural Language Processing: The Age of Transformers
#6If anyone wants to use these tools practically I urge you to have a good look at this paper : https://www.aclweb.org/anthology/P19-1439/
My take away - pretraining achieves excellent paper results but robust application is hard. There is still quite a way to go down this road for fault intolerant users and applications.
Re: Natural Language Processing: The Age of Transformers
#7If anyone wants to use these tools practically I urge you to have a good look at this paper : https://www.aclweb.org/anthology/P19-1439/ My take away - pretraining achieves excellent paper results but robust application is hard. There is still quite a way to go down this road for fault intolerant users and applications.
I'm not an expert, but after playing with some pre-trained transformers I think they are mostly good at the exact thing they're trained for. eg. GPT-2 is great for text generation, but if you try to use it for say translation, it will tend to add imagined details not in the source text. Similarly, BERT is great at sequence and token-level classification but quite bad at text generation.
Re: Natural Language Processing: The Age of Transformers
#8>Next we shall take a moment to remember the fallen heros, without whom we would not be where we are today. I am, of course, referring to the RNNs - Recurrent Neural Networks, a concept that became almost synonymous with NLP in the deep learning field.
XLNet (https://arxiv.org/abs/1906.08237) is in essence a recurrent neural network, using a transformer (which is based on neural networks) which recurrently keeps context between different batches. But the gated RNN's, such as AWD-LSTM/GRU, are fading out to the superior transformer architectures, this is true.
That's my only complain though, excellent theoretical introduction.
Although, if anyone wanted to actually implement a transformer, be ware that you want to have a 8+ GB GPU unit available, or be prepared to use cloud computing (Google Colab is free, for now). Training neural networks is quite hardware dependent still.
Re: Natural Language Processing: The Age of Transformers
#9>Next we shall take a moment to remember the fallen heros, without whom we would not be where we are today. I am, of course, referring to the RNNs - Recurrent Neural Networks, a concept that became almost synonymous with NLP in the deep learning field. XLNet ( https://arxiv.org/abs/1906.08237 ) is in essence a recurrent neural network, using a transformer (which is based on neural networks) which recurrently keeps co…
Scaleway (where the author of this post works, as I do) is a cloud service provider with a pretty interesting GPU instance: Nvidia P100 16-GB NVIDIA Tesla P100 at 1€ per hour
Re: Natural Language Processing: The Age of Transformers
#10Conversational AI is much closer than we think. Neural sequence-to-sequence models are successful in domain specific domains. But in the context of chit-chat based dialogue systems, the responses lack humanity. Undoubtedly due to the fact they don't comprehend our world. Transfer learning alleviates some of that awkwardness.
If anyone's interested in further experiments on their own. There is now a unified Python framework for dialogue models ;)