does it mean we don't need gradient descent after all to achieve the same result?
Nope, Transformer is still trained with gradient descent.
Transformer: A Novel Neural Network Architecture for Language Understanding
11–20 of 20 posts
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#12Has anyone come across good ML to do arabic-english? Seems to be a complete lack of decent training data.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#13Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering.
Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so far and additional meta data. Think customer service bots with access to customer data that can learn to interpret questions, associate it with their account information through the attention model and generate useful responses.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#14Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering. Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so f…
No doubt a holy grail for chat bots, but I'll believe it when I see it.
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#15Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering. Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so f…
No doubt a holy grail for chat bots, but I'll believe it when I see it.
"Even without evidence, everyone should believe it will solve the problem, but I won't believe it will solve the problem until there's evidence."
Is that what you just said? :)
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#16Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering. Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so f…
No doubt a holy grail for chat bots, but I'll believe it when I see it.
The harder problem, as usual, is to get enough high quality training data for a particular problem domain though.
Maybe as a POC we can try building a bot that generates relevant HN comments given the post and parent comments. Maybe I'm such a bot, how could I possibly know?
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#17Has anyone come across good ML to do arabic-english? Seems to be a complete lack of decent training data.
There's the GALE data provided by LDC (https://catalog.ldc.upenn.edu/):
- GALE Phase 1 Arabic Broadcast News Parallel Text - Part 1 (LDC2007T24) - GALE Phase 1 Arabic Broadcast News Parallel Text - Part 2 (LDC2008T09) - GALE Phase 1 Arabic Blog Parallel Text (LDC2008T02) - GALE Phase 1 Arabic Newsgroup Parallel Text - Part 1 (LDC2009T03) - GALE Phase 1 Arabic Newsgroup Parallel Text - Part 2 (LDC2009T09) - GALE Phase 2 Arabic Broadcast Conversation Parallel Text Part 1 (LDC2012T06) - GALE Phase 2 Arabic Broadcast Conversation Parallel Text Part 2 (LDC2012T14) - GALE Phase 2 Arabic Newswire Parallel Text (LDC2012T17) - GALE Phase 2 Arabic Broadcast News Parallel Text (LDC2012T18) - GALE Phase 2 Arabic Web Parallel Text (LDC2013T01)
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#18Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering. Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so f…
A "collection of words in a relative ordering" is "a sequence of words".
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#19Has anyone come across good ML to do arabic-english? Seems to be a complete lack of decent training data.
Isn't Arabic an official UN language?
Re: Transformer: A Novel Neural Network Architecture for Language Understanding
#20Very interesting approach, and intuitively it makes sense to treat language less as a sequence of words over time and more as a collection of words/tokens with meaning in their relative ordering. Now I'm wondering what would happen if a model like this were applied to different kinds of text generation like chat bots. Maybe we could build actually useful bots if they can have attention on the entire conversation so f…
They mentioned tensor2tensor, how is this related to another repo: seq2seg:https://github.com/google/seq2seq? Which one is more general?