Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

41–50 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#41

Post by codingslave »

the8472 wrote: post #39
corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

> More than that, I think NLP will unlock new ways of interacting with computers. Computers will be able to handle the ambiguity of human language, transcending their rigid “only do exactly what you tell them” models of the world. Such as google's duplex?

Yes, but duplex doesn't work and was too early.

Re: Turing-NLG: A 17B-parameter language model

#42

Post by ctoth »

p1esk wrote: post #26
corporateslave5 wrote: post #17

Earlier quoted context omitted.

I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…

What exactly do you expect from these Transformer-based models? This particular one is underwhelming because it provides a tiny improvement over the previous largest one (from Nvidia) with more than double the size. The fundamental limitation of these "optimization tools" as you call them - they don't have any common sense, and any way to query an external source of information (e.g. wikipedia), or ask a human to cla…

Some papers which you might find interesting:

Lin, B. Y., Chen, X., Chen, J., & Ren, X. (2019). KagNet: Knowledge-Aware Graph Networks for Commonsense Reasoning. 2822–2832. https://doi.org/10.18653/v1/d19-1282

Liu, W., Zhou, P., Zhao, Z., Wang, Z., Ju, Q., Deng, H., & Wang, P. (2019). K-BERT: Enabling Language Representation with Knowledge Graph. Retrieved from http://arxiv.org/abs/1909.07606

Trinh, T. H., & Le, Q. V. (2019). Do Language Models Have Common Sense? Iclr, 1–12.

Ostendorff, M., Bourgonje, P., Berger, M., Moreno-Schneider, J., Rehm, G., & Gipp, B. (2019). Enriching BERT with Knowledge Graph Embeddings for Document Classification. Retrieved from http://arxiv.org/abs/1909.08402

Re: Turing-NLG: A 17B-parameter language model

#44

Post by codingslave »

b3kart wrote: post #38
corporateslave5 wrote: post #17

Earlier quoted context omitted.

I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…

I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything ”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn…

I'm not saying this will spawn AI. Just that, like computer vision was essentially solved by CNNs, NLP will be solved by transformer models.

Re: Turing-NLG: A 17B-parameter language model

#45

Post by the8472 »

foota wrote: post #18
01100011 wrote: post #14

How long until the language models stabilize enough that we can bake them into a low-cost, low-power chip for edge uses?

I think this is largely unnecessary, can't things like TPUs handle the inference?

Putting all your speech/text onto cloud machines runs counter to e2e encrypted messaging.

Re: Turing-NLG: A 17B-parameter language model

#46

Post by [deleted] »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

[deleted]

Re: Turing-NLG: A 17B-parameter language model

#48

Post by thfuran »

codingslave wrote: post #44
b3kart wrote: post #38

Earlier quoted context omitted.

I am sorry, you come across as extremely over-enthusiastic, without too many specifics beyond “we’re just about to figure it all out”, “you just wait”, and “it’s gonna revolutionize everything ”. We’ve seen this before with ImageNet, didn’t we? When everybody thought that because ConvNets are crushing all the older methods, AI is right around the corner. Well, it turned out to be much more complicated than that, didn…

I'm not saying this will spawn AI. Just that, like computer vision was essentially solved by CNNs, NLP will be solved by transformer models.

>computer vision was essentially solved by CNNs

That is a rather contentious claim.

Re: Turing-NLG: A 17B-parameter language model

#49

Post by Barrin92 »

corporateslave5 wrote: post #17
m0zg wrote: post #10

Earlier quoted context omitted.

I think you're the one who overestimates how much this will affect NLP. I'd say bulk of what was possible to deliver with this is already here, the subsequent changes will be incremental. The cold hard truth about statistical (and by extension, deep) NLP is that it's just a fancy way of counting numbers mostly. The only way to get to _real_ language understanding is AGI, and _nobody_ is working on that. You fundament…

I respectfully disagree. I’ve been in the field full time for a few years, I watch the state of the art closely. It’s hard to see the thought/theoretical progression of deep transformer models by just reading posts here and there. I’m not saying these NLP methods will be some kind of AI, just that they will produce products, content, and ways of interacting with the world that are categorically different from what we…

>Find an answer in a text document that spans multiple non-contiguous paragraphs

>Understand context across a whole book.

to have truly much utility when it comes to context a model doesn't just need to correlate information across some text, it also needs general knowledge and understanding so that it can produce knowledge which is only implicit or not even present in the text itself.

You can make the transformers as large as you want, NLP models still fundamentally suck at answer trivial questions like "If Pete was alive in 2000 and alive in 2050, was he alive in 2025?"

Post reply on HN