Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

101–110 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#101

Post by joe_the_user »

mumblemumble wrote: post #93
joe_the_user wrote: post #58

Earlier quoted context omitted.

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols a…

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

Syntax and semantics were developed for human language, yet it's much easier to puzzle the difference in a computer language than in human language. With syntax and semantics so wrapped together, however, it kind of seems like you can go a long way with just capturing syntax, rhythm, word choice and etc. Which is to say the semantic side can be even worse than it seems, ie, nonexistent.

Re: Turing-NLG: A 17B-parameter language model

#102

Post by NeutronStar »

joe_the_user wrote: post #58
corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols a…

Back then it was noticeable after a couple words. That we are speaking about paragraphes now speaks for itself. Where will be in in 50 years?

Re: Turing-NLG: A 17B-parameter language model

#103

Post by saurkt »

Tenoke wrote: post #55
saurkt wrote: post #11

One of the team members from Project Turing. Happy to answer any questions.

Any plans on training other (non nlp) huge models using ZeRO? Specifically for Transformers - any plans to train a big model with a bigger context window? Not that this one isn't very impressive, of course.

Thanks for your kind words. Yes, we would like to next train a language representation model. And our hunch is that probably something which is a mixture of language representation and language generation would be able to get the best of both worlds.

Re: Turing-NLG: A 17B-parameter language model

#104

Post by [deleted] »

joe_the_user wrote: post #101
mumblemumble wrote: post #93

Earlier quoted context omitted.

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

Syntax and semantics were developed for human language, yet it's much easier to puzzle the difference in a computer language than in human language. With syntax and semantics so wrapped together, however, it kind of seems like you can go a long way with just capturing syntax, rhythm, word choice and etc. Which is to say the semantic side can be even worse than it seems, ie, nonexistent.

[deleted]

Re: Turing-NLG: A 17B-parameter language model

#105

Post by lowdose »

minimaxir wrote: post #79
lowdose wrote: post #50

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined. https://www.reddit.com/r/SubSimulatorGPT2/

GPT-2 X 10 is misleading; this model size is 10x, sure, but that doesn't necessarily mean the output will be 10x better. For r/SubSimulatorGPT2, it did go from 355M to 1.5B recently, but the quality isn't necessarily 4x (although it did improve). I'm more interested in shrinking models that maintain the same level of generative robustness (e.g. distillation, with distilGPT2)

The model did go from a 100 different trained models to 1. So it seems to hold at least 4x as much in knowledge but maybe we should ask the people that actually trained it.

Btw thank you for your GPT-2 simple, played around with it last weekend and it made building a toy surprisingly simple!

Re: Turing-NLG: A 17B-parameter language model

#106

Post by quotemstr »

mumblemumble wrote: post #93
joe_the_user wrote: post #58

Earlier quoted context omitted.

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols a…

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

> This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil"

Isn't that basically the same as the Winograd problem?

Re: Turing-NLG: A 17B-parameter language model

#108

Post by bonoboTP »

m0zg wrote: post #96
bonoboTP wrote: post #56

Earlier quoted context omitted.

> When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it. I don't think anyone familiar with the area thought that ConvNets will give us AGI. However, their effect has been huge! It's hard to overstate this. Computer vision used to be a small niche topic, with tons of effort required to get so…

Crucially also, convnets have exceeded human performance on several important vision tasks.

You have to be very careful with such claims. For example it may be able to tell apart tons of dog breeds at a superhuman level, but that's not really what people imagine if they hear such claims.

Also sometimes in medical imaging the conditions are very different from actual practice. For example the doctor may be worse than the convnet on certain types of low-quality, low-dynamic range images that someone preprocessed in a particular way. But sure, in the medical field some error prone, boring counting tasks and spot-the-cancer-in-your 200th-image-today, the machine can perform actually better.

But what tasks specifically do you have in mind?

Re: Turing-NLG: A 17B-parameter language model

#109

Post by rjeli »

I have been bearish on AGI, but GPT2 surprised me with the lucidity of its samples.

My take from the past few years is that we're 99% done with the visual cortex - convolutional nets can be trained to perform any visual task a human can in edit: it's possible cognition follows from language, which would be convenient. is GPT2 smarter than a dog? I don't think so but I could be wrong ¯\_(ツ)_/¯

Re: Turing-NLG: A 17B-parameter language model

#110

Post by kragen »

rjeli wrote: post #109

I have been bearish on AGI, but GPT2 surprised me with the lucidity of its samples. My take from the past few years is that we're 99% done with the visual cortex - convolutional nets can be trained to perform any visual task a human can in edit: it's possible cognition follows from language, which would be convenient. is GPT2 smarter than a dog? I don't think so but I could be wrong ¯\_(ツ)_/¯

I have been bearish on AGI, but GPT2 surprised me with the lucidity of both paths. I still maintain my support for the basic metric of the GPT-I. However, I have a number of requirements on how my proposal is to be funded to resolve concerns. First, I strongly believe that academic research should be the method of choice (that is, if we are to figure out how to make AGI possible), and I advocate funding to support results from the central bank community. Second, given that the GPT can be articulated in mathematical terms, this should be reflected in funding policy. A very serious concern is that if funding of GPT is disincentivized, investors may react similarly to the way they reacted to AGI. This is
Post reply on HN