Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

141–149 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#141

Post by dlkf »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

> So the model is bounded by the context window, which is currently mostly at 512 tokens, and is thus, bounded in how much it can understand.

This is begging the question as to whether the model "understands" anything at all. And once you adopt a definition of "understanding" that isn't equivalent to "got a high score on some pointless academic challenge" the answer is a resounding "no." The whole enterprise of AGI hype is based on this equivocation of words like "understanding" and "intelligence." We use a very restricted definition in proving that the tech is smart, and then switch out our restricted definition for the colloquial one when the audience isn't looking.

> This will unlock better conversational abilities

Shouldn't be hard given that as it stands there are none, except for creating a human-sounding slurry that is devoid of real content.

> but also, better ways to understand how different pieces of textual information relate

Is this a real need? What problem does this solve that forums + wikipedia + arxiv + google + a literate human hasn't already?

Re: Turing-NLG: A 17B-parameter language model

#143

Post by ThinkCritically »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

To any of the readers out there, I am currently looking into better understanding the potential business applications of this NLP technology. It would be especially interesting to hear from people who work extensively in social media as to how "Automated reading" could one day help them with their daily job.

If you have time to discuss, DM me on twitter - I am @ralphbrooks.

Re: Turing-NLG: A 17B-parameter language model

#144

Post by elcomet »

mumblemumble wrote: post #93
joe_the_user wrote: post #58

Earlier quoted context omitted.

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one. Human language is this marvelous framework involving symbols a…

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

I just tried a few examples and it always returned the good word to me.

I'm going home -> Je rentre a la maison

Home sweet home -> La douceur du foyer (the translation is weird but the word foyer was expected)

I feel at home -> Je me sens chez moi (this one is particularly good, it didn't translate the word home directly)

Can you share your exemples where it fails?

Re: Turing-NLG: A 17B-parameter language model

#145

Post by mumblemumble »

elcomet wrote: post #144
mumblemumble wrote: post #93

Earlier quoted context omitted.

> So, are reasonable examples now of these models allowing semantic context? This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil." Right now it seems to very frequently choose "accueil", which is generally wrong, except whe…

I just tried a few examples and it always returned the good word to me. I'm going home -> Je rentre a la maison Home sweet home -> La douceur du foyer (the translation is weird but the word foyer was expected) I feel at home -> Je me sens chez moi (this one is particularly good, it didn't translate the word home directly) Can you share your exemples where it fails?

Not offhand. It's a problem I've noticed more often when looking at translations of larger bodies of text, though. Translating simple, highly standardized sentences like those is what neural translation does best, because it is able to basically just consult an internal phrasebook that it's compiled from its training data.

Re: Turing-NLG: A 17B-parameter language model

#146

Post by amylene »

p1esk wrote: post #51
sanxiyn wrote: post #36

Earlier quoted context omitted.

> they don't have any common sense What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast... Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi. It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't tha…

Try asking it a follow up question not commonly found in the training data: such as "do you think bulgogi would grow on Mars?", and see what kind of gibberish you will get in response. Moreover, the model has no way of self-diagnosing whenever it produces gibberish.

What is the baseline comparison here? If you asked a random 100 people on the street this question?

Re: Turing-NLG: A 17B-parameter language model

#147

Post by perl4ever »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

Let's say you had a problem stated as follows:

You have several thousand documents in HTML, with roughly similar content (to a human) but not entirely consistent in the ordering of the sections, or the formatting, or the names, or the language and structure. But each document describes an entity of the same class (to a human) and very nearly all of them have a section that summarizes the document. They have a lot of parts in common, but they are fundamentally not designed to line up with a structure for data processing.

Is there any practical way to find that summary? Sure, something obvious that takes no time to script is to look for something like "Overview" but you quickly get bogged down in exceptions.

Maybe this seems very mundane and simpleminded, but it is the sort of thing that many people would assume is best done by a human. Is there anything current or in the near future that would be significantly easier than a human reading all of them?

Re: Turing-NLG: A 17B-parameter language model

#148

Post by perl4ever »

mumblemumble wrote: post #112
quotemstr wrote: post #106

Earlier quoted context omitted.

> This is about where I am stuck. I'll start believing that we truly are on the cusp of a revolution as soon as I see Google Translate reliably knowing when to translate "home" into French as "domicile", "foyer", something those lines, or as "accueil" Isn't that basically the same as the Winograd problem?

I would guess that it's a bit easier. With what I was proposing, you just need to be able to infer the semantics of certain words. With the Winograd problem, you need to grasp the semantics of all the words, and then use that knowledge to infer a pronoun's antecedent based on what yields a more sensical overall interpretation of the sentence.

"use that knowledge to infer a pronoun's antecedent based on what yields a more sensical overall interpretation of the sentence"

People often assume a very benign, civilized environment for AI. In ordinary life, human beings (on the internet or off) take an adversarial approach to other people modeling the logical framework behind an utterance. They either subvert it for amusement (trolling or comedy) or profit (politics, propaganda, sales), and a tremendous amount of effort goes into it, much of which is very effective.

As people have observed, it can be very easy to convince a human that a machine is intelligent, as with ELIZA. But if you violate the presumptions of trust that a machine is designed with, it's going to be surprisingly vulnerable. To be superior to humans, a machine would have to be able to fend off an intelligent person trying to undermine it, not just work when it is spoon-fed.

Re: Turing-NLG: A 17B-parameter language model

#149

Post by jamiek88 »

lowdose wrote: post #50

This does GPT-2 X 10. For anyone wondering what GPT-2 is doing look at this baffling subreddit and marvel at how one GPT-2 model trained for $70k spits out better comedy than everybody on the payroll of Netflix combined. https://www.reddit.com/r/SubSimulatorGPT2/

https://www.reddit.com/r/SubSimulatorGPT2/comments/f7dlec/do...

That thread title and posts were stunning!

This is a great niche for GPT-2

Post reply on HN