Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

111–120 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#111

Post by Someone »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

”People are vastly underestimating the changes that are about to come from NLP.”

Maybe, maybe not. IMO, we don’t know what problem we have to solve to get what you describe. We also don’t have a metric as to how far we are along the path towards that goal (do we need 100B parameters? A trillion?), nor do we have any idea as to whether the current approach can get us there.

Re: Turing-NLG: A 17B-parameter language model

#112

Post by mumblemumble »

quotemstr wrote: post #106
mumblemumble wrote: post #93

Earlier quoted context omitted.

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

> This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil" Isn't that basically the same as the Winograd problem?

I would guess that it's a bit easier. With what I was proposing, you just need to be able to infer the semantics of certain words. With the Winograd problem, you need to grasp the semantics of all the words, and then use that knowledge to infer a pronoun's antecedent based on what yields a more sensical overall interpretation of the sentence.

Re: Turing-NLG: A 17B-parameter language model

#113

Post by bonoboTP »

kragen wrote: post #110
rjeli wrote: post #109

I have been bearish on AGI, but GPT2 surprised me with the lucidity of its samples. My take from the past few years is that we're 99% done with the visual cortex - convolutional nets can be trained to perform any visual task a human can in edit: it's possible cognition follows from language, which would be convenient. is GPT2 smarter than a dog? I don't think so but I could be wrong ¯\_(ツ)_/¯

I have been bearish on AGI, but GPT2 surprised me with the lucidity of both paths. I still maintain my support for the basic metric of the GPT-I. However, I have a number of requirements on how my proposal is to be funded to resolve concerns. First, I strongly believe that academic research should be the method of choice (that is, if we are to figure out how to make AGI possible), and I advocate funding to support re…

You generated this with GPT, right?

Re: Turing-NLG: A 17B-parameter language model

#115

Post by saurkt »

slashcom wrote: post #99
01100011 wrote: post #14

How long until the language models stabilize enough that we can bake them into a low-cost, low-power chip for edge uses?

There’s lots of work on distillation, smaller models, approximations, etc. People already have simpler forms of these running on smartphones. Models seem to be growing faster than we can make them small though :D

Yes, these things keep us up at night as well :-).

Re: Turing-NLG: A 17B-parameter language model

#116

Post by sailingparrot »

riku_iki wrote: post #100
sailingparrot wrote: post #92

Earlier quoted context omitted.

* BERT & language representation models: They basically turn a sentence into a compact vector that represents it so you can then do some downstream task on it such as sentiment detection, or matching the similarity between two sentences etc. * GPT & language generation models: Given some context (say a sentence), they can generate text to complete it, or to summarize it, etc. The task here is to actually write someth…

Both are language representation models, text generation is just a way of training model. BERT is also trained on text generation task: it asked to fill gaps in text (15% of text is blanked during training).

Maybe I am not understanding your point.

Out of the box, given a sequence of n tokens, BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary. You can then fine-tune a model on this representation to do various tasks, e.g. sentiment classification. Thus BERT is said to be a language representation model.

Out of the box, given a sequence, GPT-2 returns a distribution over the vocabulary [2] from which you can draw to find the most likely next word. Thus GPT-2 is said to be a language generation model.

You could of course play with the masking token of BERT call it recursively to force BERT to generate something, and you could chop off some layers of GPT-2 to get some representation of your input sequence, but I think that is a little past the original question.

[1] https://github.com/google-research/bert/blob/master/modeling...

[2] https://github.com/openai/gpt-2/blob/master/src/model.py#L17...

Re: Turing-NLG: A 17B-parameter language model

#117

Post by riku_iki »

sailingparrot wrote: post #116
riku_iki wrote: post #100

Earlier quoted context omitted.

Both are language representation models, text generation is just a way of training model. BERT is also trained on text generation task: it asked to fill gaps in text (15% of text is blanked during training).

Maybe I am not understanding your point. Out of the box, given a sequence of n tokens, BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary. You can then fine-tune a model on this representation to do various tasks, e.g. sentiment classification. Thus BERT is said to be a language representation model. Out of the box, given a sequence, GPT-2 returns…

> BERT returns a tensor of dimension (n_tokens, hidden_size) [1]. Where hidden size has no relationship with the vocabulary

"BERT returns" is ambiguous here. During pretraining last layer is loggits for one hot vocab vector, the same as in GPT: https://github.com/google-research/bert/blob/master/run_pret...

Re: Turing-NLG: A 17B-parameter language model

#118

Post by kragen »

bonoboTP wrote: post #113
kragen wrote: post #110

Earlier quoted context omitted.

I have been bearish on AGI, but GPT2 surprised me with the lucidity of both paths. I still maintain my support for the basic metric of the GPT-I. However, I have a number of requirements on how my proposal is to be funded to resolve concerns. First, I strongly believe that academic research should be the method of choice (that is, if we are to figure out how to make AGI possible), and I advocate funding to support re…

You generated this with GPT, right?

Yes.

Re: Turing-NLG: A 17B-parameter language model

#119

Post by lopmotr »

skywhopper wrote: post #29
codingslave wrote: post #19

Earlier quoted context omitted.

Replying on my laptop. Basically, transformer models are the best for NLP. They use something called attention based mechanisms, which allows the model to draw correlations between pieces of text/tokens that are far apart. The issue is that this is an O(n^2) operation. So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand. Recent inn…

If this is really where the researchers think these tools are headed (and I don't really doubt you on that point), then this is incredibly dangerous stuff. No matter how good your system is, the impact of implicit, unintentional, and non-targeted bias is huge on the sorts of content these systems will produce. But expose it to the levels of intentional manipulation present on the Internet of today, and these models d…

Google search already faces that problem. Is that unsafe too?

Re: Turing-NLG: A 17B-parameter language model

#120

Post by the8472 »

joe_the_user wrote: post #97
lowdose wrote: post #50

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined. https://www.reddit.com/r/SubSimulatorGPT2/

Sure, this shows robots good at more or less fitting the formula for various types of typical articles or posts. Still, things that make no sense accumulate over sentence or paragraphs, depending on how formulaic a given format might be. So this simultaneously generally impressive but not useful for any one thing.

Perhaps it could be used to improve autocomplete keyboards. The more words it can predict in advance the fewer keystrokes/taps are needed to convey thoughts
Post reply on HN