Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

91–100 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#91

Post by octbash »

igravious wrote: post #57
saurkt wrote: post #32

Earlier quoted context omitted.

SQUAD and GLUE are tasks for language representation models -- aka BERT-like. This is a language generation model -- GPT-like. Hence, SQUAD/GLUE test sets are not really applicable. We are reporting on the wikitext and lambada sets that openAI also uses for similar models (numbers are in the blogpost).

What's the difference between the two models?

One is a language generation model, the other is a fill-in-the-blank model. It sounds like they might be similar, but in practice they are different enough objectives (and in particular the "bi-directional" aspect of BERT-type models) that the models learn different things.

Re: Turing-NLG: A 17B-parameter language model

#92

Post by sailingparrot »

igravious wrote: post #57
saurkt wrote: post #32

Earlier quoted context omitted.

SQUAD and GLUE are tasks for language representation models -- aka BERT-like. This is a language generation model -- GPT-like. Hence, SQUAD/GLUE test sets are not really applicable. We are reporting on the wikitext and lambada sets that openAI also uses for similar models (numbers are in the blogpost).

What's the difference between the two models?

* BERT & language representation models: They basically turn a sentence into a compact vector that represents it so you can then do some downstream task on it such as sentiment detection, or matching the similarity between two sentences etc.

* GPT & language generation models: Given some context (say a sentence), they can generate text to complete it, or to summarize it, etc. The task here is to actually write something.

Re: Turing-NLG: A 17B-parameter language model

#93

Post by mumblemumble »

joe_the_user wrote: post #58
corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols a…

> So, are reasonable examples now of these models allowing semantic context?

This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil."

Right now it seems to very frequently choose "accueil", which is generally wrong, except when you're talking about websites and software user interfaces. That it's biased so strongly toward that error speaks volumes about how critical semantics are to sorting out natural language, and also about how bad current NLP systems are at dealing with semantics.

Re: Turing-NLG: A 17B-parameter language model

#94

Post by bitL »

lowdose wrote: post #50

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined. https://www.reddit.com/r/SubSimulatorGPT2/

"What's the best way to kill a reddit thread?

Just make it a year long thread and wait for the year to end." -- circlejerkGPT2Bot

Re: Turing-NLG: A 17B-parameter language model

#96

Post by m0zg »

bonoboTP wrote: post #56
b3kart wrote: post #38

Earlier quoted context omitted.

I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything ”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn…

> When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it. I don't think anyone familiar with the area thought that ConvNets will give us AGI. However, their effect has been huge! It's hard to overstate this. Computer vision used to be a small niche topic, with tons of effort required to get so…

Crucially also, convnets have exceeded human performance on several important vision tasks.

Re: Turing-NLG: A 17B-parameter language model

#97

Post by joe_the_user »

lowdose wrote: post #50

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined. https://www.reddit.com/r/SubSimulatorGPT2/

Sure, this shows robots good at more or less fitting the formula for various types of typical articles or posts. Still, things that make no sense accumulate over sentence or paragraphs, depending on how formulaic a given format might be. So this simultaneously generally impressive but not useful for any one thing.

Re: Turing-NLG: A 17B-parameter language model

#99

Post by slashcom »

01100011 wrote: post #14

How long until the language models stabilize enough that we can bake them into a low-cost, low-power chip for edge uses?

There’s lots of work on distillation, smaller models, approximations, etc. People already have simpler forms of these running on smartphones. Models seem to be growing faster than we can make them small though :D

Re: Turing-NLG: A 17B-parameter language model

#100

Post by riku_iki »

sailingparrot wrote: post #92
igravious wrote: post #57

Earlier quoted context omitted.

What's the difference between the two models?

* BERT & language representation models: They basically turn a sentence into a compact vector that represents it so you can then do some downstream task on it such as sentiment detection, or matching the similarity between two sentences etc. * GPT & language generation models: Given some context (say a sentence), they can generate text to complete it, or to summarize it, etc. The task here is to actually write someth…

Both are language representation models, text generation is just a way of training model. BERT is also trained on text generation task: it asked to fill gaps in text (15% of text is blanked during training).
Post reply on HN