One of the team members from Project Turing. Happy to answer any questions.
Have you evaluated against the AI2 Leaderboard benchmarks? https://leaderboard.allenai.org/
Not yet. We will try to run against those benchmarks soon.
Turing-NLG: A 17B-parameter language model
31–40 of 149 posts
Re: Turing-NLG: A 17B-parameter language model
#32One of the team members from Project Turing. Happy to answer any questions.
Why the lack of number on the more popular SQuAD and Glue benchmarks?
SQUAD and GLUE are tasks for language representation models -- aka BERT-like. This is a language generation model -- GPT-like. Hence, SQUAD/GLUE test sets are not really applicable. We are reporting on the wikitext and lambada sets that openAI also uses for similar models (numbers are in the blogpost).
Re: Turing-NLG: A 17B-parameter language model
#33Earlier quoted context omitted.
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
I'm a researcher in the field. Not in NLP anymore, but I worked on that as well, years ago, and I keep up with the research. You can't "understand the context" of "War and Peace" unless you have real, actual AGI. I doubt actually it can be fully understood at all when translated to English and read by someone without the right cultural background. This is an extreme example, chosen to make it easy to see that it appl…
Maybe. I take a really optimistic view of attention based mechanisms, like I explained and added to my original post. If you read the recent reformer paper, they produce a new way of computation to start building a model that can in some way, encode the relationships between different parts of war and piece. The bottle neck right now is computation, we don’t know how well these models can learn when that bottle neck is removed!
I’m optimistic because I believe that the contextual information that you are describing, is already there in the vast expanse of the internet.
But I will also add, I think none of this will spawn AI, just that it will spawn new technologies that are categorically different.
Re: Turing-NLG: A 17B-parameter language model
#34B = Billion, not Byte. For second I was like, WTF?
I thought I am the only one. There is no trigger to deep learning.
But the article is fascinating nevertheless. Not sure is alphago breakthrough.
Re: Turing-NLG: A 17B-parameter language model
#35Earlier quoted context omitted.
There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe. For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sen…
any books/papers that address "goal" oriented NLP or hybrid based discussion?
There's interesting research out of Deepmind on two parts of this, which are using the Transformer model in Reinforcement Learning contexts[0] and creating textual GANs[1].
As you are probably aware, GANs are one of the important tools that have driven forward image synthesis and until recently it was impossible to apply them to text, so I expect this to push us quite a bit forward. There's also ongoing work in the selection of the metric to use to evaluate the generated text, and discriminate between human and machine-generated text.
Re: Turing-NLG: A 17B-parameter language model
#36Earlier quoted context omitted.
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size. The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to cla…
> they don't have any common sense
What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast...
Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi.
It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't that common sense? If it isn't, what do you mean by "common sense"?
Re: Turing-NLG: A 17B-parameter language model
#37One of the team members from Project Turing. Happy to answer any questions.
How does it compare to Google’s BERT and do you have an online demo? Here’s a demo of BERT https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
(Similar to the response for another question.)
BERT is a language representation model while Turing-NLG is a language generation model (similar to GPT). They are not directly comparable (they can potentially be massaged to mimic the other, but, not something that we have done yet.)
Re: Turing-NLG: A 17B-parameter language model
#38Earlier quoted context omitted.
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it. Transformers are great (well, if you have the compute that is) don’t get me wrong, but let’s not get ahead of ourselves. The field is over-hyped as it is.
Re: Turing-NLG: A 17B-parameter language model
#39People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
> More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world.
Such as google's duplex?
Re: Turing-NLG: A 17B-parameter language model
#40People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create,…
Replying from my laptop.
I agree, there needs to be a way to represent relationships between information. I personally don't think knowledge graphs will be the ones to do it, not because they dont work, but because of how imperfect they are, in the data quality sense.
See this paper here:
"DIFFERENTIABLE REASONING OVER A VIRTUAL KNOWLEDGE BASE" https://openreview.net/pdf?id=SJxstlHFPH
Which is a recent effort, among many, by google research to build a model that can view a document as a knowledge graph, instead of explicitly tying pieces of the document to the graph, the idea to is to create a graph from the document. This is paper is a bit different from that, they do input a knowledge graph for training, but I think the idea and track of where they are headed has a ton of room to evolve. The trick is that transformer models have unlocked the ability to understand the text, so all of this "quasi knowledge graph extraction" that i was just explaining, is only recently possible! There's no research on it, because the baseline understanding of tokens has been too primitive. This is why there is so much room to grow, BERT has unlocked new methods, it can be used as a base for a ton of new NLP.
Just to emphasize again, I'm not saying what I outlined above will be a good way to do it, just that ideas like this could only be tested recently. There's a million new ways to spin this problem.