Turing-NLG: A 17B-parameter language model
microsoft.com
Turing-NLG: A 17B-parameter language model
1–10 of 149 posts
Re: Turing-NLG: A 17B-parameter language model
#2Those summaries look impressive, although a bit repepetive
Re: Turing-NLG: A 17B-parameter language model
#3People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP, and with it, the ability to create, synthesize, and understand content, will change the internet.
More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world.
Edit:
Adding this to give more technical context. I think most people don’t know where the line is currently between what possible, and what’s not, but also what we are on the cusp of. And we are on the cusp of a lot.
A quick explanation of one area is here:
Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent innovations, and further study, will broaden the context window, and thus unlock better reading comprehension and context understanding. For instance, the ability to answer a question using a piece of text is mostly stuck at just finding one paragraph. The future will see models that can find multiple different paragraphs, understand how they relate, pull the relevant information, and synthesize it. This sounds like a minor step forwards, but its important. This will unlock better conversational abilities, but also, better ways to understand how different pieces of textual information relate. The scattershot of information across the internet can go away. Computers can better understand context to act on human intention through language, unlocking the ability to handle ambiguity. This will change the internet.
Again to empathize, these models only started showing up in 2017! The progress has been rapid.
Re: Turing-NLG: A 17B-parameter language model
#4People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I can't wait for the day we see "deep-dream" styled literary works.
Re: Turing-NLG: A 17B-parameter language model
#5People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I can't wait for the day we see "deep-dream" styled literary works.
There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe.
For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sense.
Re: Turing-NLG: A 17B-parameter language model
#6Earlier quoted context omitted.
I can't wait for the day we see "deep-dream" styled literary works.
There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe. For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sen…
any books/papers that address "goal" oriented NLP or hybrid based discussion?
Re: Turing-NLG: A 17B-parameter language model
#7Those summaries look impressive, although a bit repepetive
I see what you did there?
Re: Turing-NLG: A 17B-parameter language model
#8People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
If what you claim is true, it's not just the internet that will be changed by NLP but all of civilization!
Re: Turing-NLG: A 17B-parameter language model
#9Earlier quoted context omitted.
There have been plenty of "AI written" textual works, like the AI Dungeon for a D&D game, or the AI Recipe Generator that attempts to make something that looks like a cooking recipe. For the most part they aren't successful because the "AI" isn't smart enough to have a goal in mind, so they end up just monkey-cheesing everything. Pasting together snippets in ways that are usually grammatically correct but make no sen…
any books/papers that address "goal" oriented NLP or hybrid based discussion?
The ability to set your own goals and task yourself to achieve them is the essence of AI. Not "AI" as we know it today, but Sci-fi AI where it's a machine person.
Re: Turing-NLG: A 17B-parameter language model
#10People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental.
The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundamentally cannot interact comfortably with a human if your system does not have probabilistic, contextualized cognition, and can't incorporate knowledge about the world.