Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

51–60 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#51

Post by p1esk »

sanxiyn wrote: post #36
p1esk wrote: post #26

Earlier quoted context omitted.

What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size. The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to cla…

> they don't have any common sense What do you mean by this? Of course they do, learned from their training data. For example, here is quote from conversation 38 of https://github.com/google-research/google-research/blob/mast... Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi. It seems to me Meena "knows" bulgogi and kimchi are Korean foods. Isn't tha…

Try asking it a follow up question not commonly found in the training data: such as "do you think bulgogi would grow on Mars?", and see what kind of gibberish you will get in response. Moreover, the model has no way of self-diagnosing whenever it produces gibberish.

Re: Turing-NLG: A 17B-parameter language model

#52

Post by Tenoke »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

> mostly at 512 token

GPT uses 1024 tokens context window which does work out to a fair amount (given the massive vocab of 50k+ which means a token can be more than a word), though of course it's pretty limited.

Google's recent Reformer[0] allows you to do attention much more cheaply and I'm currently training a Reformer model that isn't quite as big but has a context of ~64k tokens (though a much smaller vocab). I'm not completely sure if this is the solution but it looks like a step in that direction and so far the model is doing pretty good (I also plan to post the weights when I'm finished, though I am not sure if Google don't just plan to do that themselves).

I am somewhat disappointed they went with just 1024 for this model, too though.

0. https://ai.googleblog.com/2020/01/reformer-efficient-transfo...

Re: Turing-NLG: A 17B-parameter language model

#53

Post by corporateslave5 »

Tenoke wrote: post #52
corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

> mostly at 512 token GPT uses 1024 tokens context window which does work out to a fair amount (given the massive vocab of 50k+ which means a token can be more than a word), though of course it's pretty limited. Google's recent Reformer[0] allows you to do attention much more cheaply and I'm currently training a Reformer model that isn't quite as big but has a context of ~64k tokens (though a much smaller vocab). I'm…

Yep, the number has been expanding. I just picked 512 because Bert uses that. But yeah 50k and beyond unlocks new ways of using context. I can’t wait to see how they optimize and improve the reformer model.

I expect there to be an improvement like there was for

BERT -> Albert

Re: Turing-NLG: A 17B-parameter language model

#55

Post by Tenoke »

saurkt wrote: post #11

One of the team members from Project Turing. Happy to answer any questions.

Any plans on training other (non nlp) huge models using ZeRO?

Specifically for Transformers - any plans to train a big model with a bigger context window?

Not that this one isn't very impressive, of course.

Re: Turing-NLG: A 17B-parameter language model

#56

Post by bonoboTP »

b3kart wrote: post #38
corporateslave5 wrote: post #17

Earlier quoted context omitted.

I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…

I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything ”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn…

> When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn’t it.

I don't think anyone familiar with the area thought that ConvNets will give us AGI.

However, their effect has been huge! It's hard to overstate this. Computer vision used to be a small niche topic, with tons of effort required to get something working even on simple images. The quality of today's ConvNet predictions is way beyond anybody's imagination in around 2010. Models built around that time were like a house of cards. Extremely carefully crafted for specific scenarios, where moving one threshold a bit would destroy your output.

Re: Turing-NLG: A 17B-parameter language model

#57

Post by igravious »

saurkt wrote: post #32
osipov wrote: post #13

Earlier quoted context omitted.

Why the lack of number on the more popular SQuAD and Glue benchmarks?

SQUAD and GLUE are tasks for language representation models -- aka BERT-like. This is a language generation model -- GPT-like. Hence, SQUAD/GLUE test sets are not really applicable. We are reporting on the wikitext and lambada sets that openAI also uses for similar models (numbers are in the blogpost).

What's the difference between the two models?

Re: Turing-NLG: A 17B-parameter language model

#58

Post by joe_the_user »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world.

So, are reasonable examples now of these models allowing semantic context? So, far, what I have seen is generated text where the lack of understanding takes three paragraphs to become obvious rather than one.

Human language is this marvelous framework involving symbols associating with other symbols as well as to well-known and vaguely-guessed facts about the world.

Human relations are very robust and, for example, two people can have a longish conversations where at the end, they realize they're talking about two different people (or different days or events). But in those circumstances, they can correct and adjust. "Solid" understanding is there but it's under a lot of layers of social cues and protocols and multiple meanings.

Re: Turing-NLG: A 17B-parameter language model

#59

Post by lostcolony »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

>More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world.

As exciting as this sounds, I can't help but feel that given -we- haven't figured out how to handle the ambiguity of human language, I'm not convinced a computer attempting to is really markedly better for many use cases than requiring exactness. But operating at a human level of 'understanding', and being broadly accessible, may be enough to change the world. Hopefully for the better.

Re: Turing-NLG: A 17B-parameter language model

#60

Post by joe_the_user »

jandrese wrote: post #9
okigan wrote: post #6

Earlier quoted context omitted.

any books/papers that address "goal" oriented NLP or hybrid based discussion?

The ability to set your own goals and task yourself to achieve them is the essence of AI. Not "AI" as we know it today, but Sci-fi AI where it's a machine person.

Indeed, but that kind of describes the gulf between current language processing and present AI. Present AI generates tokens that seem to have meaning or seem like an appropriate response to a statement but where it become evident after 2-3 paragraphs, there's no substantial relation to either underlying meaning or underlying goals.

Part of this is "underlying meaning" is an intuitive way to describe things but whatever is underlying here is more tenuous than a classical logic/GOFAI model of the world but more "solid" than a long, clever stream of associations.

Post reply on HN