B = Billion, not Byte. For second I was like, WTF?
Turing-NLG: A 17B-parameter language model
21–30 of 149 posts
Re: Turing-NLG: A 17B-parameter language model
#22One of the team members from Project Turing. Happy to answer any questions.
How does it compare to Google’s BERT and do you have an online demo?
Here’s a demo of BERT https://www.pragnakalp.com/demos/BERT-NLP-QnA-Demo/
Re: Turing-NLG: A 17B-parameter language model
#23Earlier quoted context omitted.
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
Since you seem to be observing this closely, I'm curious what steps are being taken to identify and avoid bias in the results generated by these systems?
Re: Turing-NLG: A 17B-parameter language model
#24People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…
I would agree with you to a certain extent, but I still think there is a big missing component to make this a reality. Larger and more accurate general language models are great, but to enable use cases other than categorization, translation, summarization, etc., there will almost certainly have to a be contextualized knowledge graph layer. This is basically what I assume you mean when you say "the ability to create, synthesize, and understand content." The way I see it, transformer-based general language models will be the first and last step of a NLP system. I other words, they will do the raw processing of the input and will do the "take this output and put it in natural language based on this context" portion. Automating the part in the middle, the part that actually understands what the text means and can do logic with it, there's still no progress on that AFAIK.
This is similar to how computer vision models work great in most cases, but to build a self-driving car you still need all the other components that do path planning, predicting what the cars around you will do based on the state of the environment, etc.
Re: Turing-NLG: A 17B-parameter language model
#25One of the team members from Project Turing. Happy to answer any questions.
Have you evaluated against the AI2 Leaderboard benchmarks? https://leaderboard.allenai.org/
Re: Turing-NLG: A 17B-parameter language model
#26Earlier quoted context omitted.
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size.
The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to clarify.
Another big problem is we don't have any way to do quality filtering on the outputs. From my experiments with GPT-2, it produces one interesting paragraph of text out of 20 - if you squint at it really hard. And most of those 20 don't make much sense at all.
So no, the existing ideas are definitely not enough. Maybe some novel hybrid of symbolic AI with statistical optimization will lead to a breakthrough. This one does not strike me as anything other than "let's use moar weights!!"
Re: Turing-NLG: A 17B-parameter language model
#27unfortunately they abstained from participation in more popular SQuAD and Glue benchmarks..
[deleted]
Re: Turing-NLG: A 17B-parameter language model
#28Earlier quoted context omitted.
I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…
I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…
I'm a researcher in the field. Not in NLP anymore, but I worked on that as well, years ago, and I keep up with the research. You can't "understand the context" of "War and Peace" unless you have real, actual AGI. I doubt actually it can be fully understood at all when translated to English and read by someone without the right cultural background. This is an extreme example, chosen to make it easy to see that it applies to any non-trivial text.
Let's take a question answering example. Take just about any recent deep learning paper and try to answer detailed, higher level questions against it. To use a concrete example, take MobileNet V3 paper and ask your system "do I put activation before or after squeeze and excitation" (correct answer is "before"), or "do I need a bias in squeeze and excitation module" (correct answer is "it depends on the task"). You won't be able to, because a lot of things are just _assumed_, just like in any other realistic example of text written for human consumption. The facts are encoded externally as information about the world, and they're so fine grained and contextual, that we don't even know how to begin incorporating them into the answers, let alone do so contextually and probabilistically, like human mind does.
Re: Turing-NLG: A 17B-parameter language model
#29Earlier quoted context omitted.
Can you explain your reasoning? It's easy to imagine a new invention or idea will revolutionize everything because it's never been done before and feels powerful, but even if it works, it might not. It sounds like you believe NLP will enable more general voice control of computers than Siri/Alexa/etc. But will that really be much more significant than people expect? Google is already pretty good at understanding ambi…
Replying on my laptop. Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent inn…
If this is really where the researchers think these tools are headed (and I don't really doubt you on that point), then this is incredibly dangerous stuff. No matter how good your system is, the impact of implicit, unintentional, and non-targeted bias is huge on the sorts of content these systems will produce. But expose it to the levels of intentional manipulation present on the Internet of today, and these models don't stand a chance of producing something that safely does what you claim.
I'm happy to be wrong about this, but I'm not seeing any discussion about the safety and security of using these systems. And if it's not even being discussed, we can be sure nothing's actually being done about it. Selling promises of active-agent computers interpreting human intent and summarizing information from the Internet without addressing this concern is irresponsible at this point.