One of the team members from Project Turing. Happy to answer any questions.
Turing-NLG: A 17B-parameter language model
11–20 of 149 posts
Re: Turing-NLG: A 17B-parameter language model
#12unfortunately they abstained from participation in more popular SQuAD and Glue benchmarks..
Re: Turing-NLG: A 17B-parameter language model
#13One of the team members from Project Turing. Happy to answer any questions.
Why the lack of number on the more popular SQuAD and Glue benchmarks?
Re: Turing-NLG: A 17B-parameter language model
#14How long until the language models stabilize enough that we can bake them into a low-cost, low-power chip for edge uses?
Re: Turing-NLG: A 17B-parameter language model
#15People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I can't wait for the day we see "deep-dream" styled literary works.
But what would be the point... after the novelty wears off, no one considers deep dream anything other than an outcome of the system, not as art with an intended meaning (unless the point of art without meaning)
Re: Turing-NLG: A 17B-parameter language model
#16People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
Can you explain your reasoning? It's easy to imagine a new invention or idea will revolutionize everything because it's never been done before and feels powerful, but even if it works, it might not. It sounds like you believe NLP will enable more general voice control of computers than Siri/Alexa/etc. But will that really be much more significant than people expect? Google is already pretty good at understanding ambiguous queries. It's eliminated the need to know where to look for information. Or are you talking about the computer writing code or doing business itself?
Re: Turing-NLG: A 17B-parameter language model
#17People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there.
I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we have seen in the past.
For instance, question and answering tasks have only recently been able to:
Find an answer in a text document that spans multiple non contiguous paragraphs
Understand context across a whole book.
The context window of current nlp is stuck at 512 tokens, mostly because of computational complexity. This has been broken just recently by the reformer model. Which is a primitive, early way to get around the computation costs of attention mechanisms.
Just wait. The ideas are there. They just take time to refine.
Re: Turing-NLG: A 17B-parameter language model
#18How long until the language models stabilize enough that we can bake them into a low-cost, low-power chip for edge uses?
I think this is largely unnecessary, can't things like TPUs handle the inference?
Re: Turing-NLG: A 17B-parameter language model
#19People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
Can you explain your reasoning? It's easy to imagine a new invention or idea will revolutionize everything because it's never been done before and feels powerful, but even if it works, it might not. It sounds like you believe NLP will enable more general voice control of computers than Siri/Alexa/etc. But will that really be much more significant than people expect? Google is already pretty good at understanding ambi…
Replying on my laptop.
Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand.
Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding.
For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important.
This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet.
Re: Turing-NLG: A 17B-parameter language model
#20People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
Can you explain more about what you mean by "able to handle the ambiguity of human language"? Such a claim goes far beyond language processing itself. Such a capability would have to involve cultural and interpersonal dynamics, domain knowledge, and definitive agency on the part of the computer.
And all of _that_ is just to interpret sincere, honest attempts at communication. Can it safely and appropriately handle humor, irony, and sarcasm? What about coordinated malicious attacks?