Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Turing-NLG: A 17B-parameter language model

microsoft.com

Post reply on HN
Print view View original

131–140 of 149 posts

Re: Turing-NLG: A 17B-parameter language model

#131

Post by jaggednad »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

I think you’re right that we are on the cusp or real natural language understanding. It’s an incredible moment. But we are also kinda far from these huge models being in every home because of the enormous amount of compute they require. Even running inference with these things is kinda expensive

Re: Turing-NLG: A 17B-parameter language model

#132

Post by sgt101 »

p1esk wrote: post #89
sgt101 wrote: post #74

Earlier quoted context omitted.

I'd buy that if Meena could infer and reason about her own answers. Human: Do you like Korean food in general? Meena: It's okay. I like beef bulgogi, but I'm not a huge fan of kimchi. Human: Ok what should I shop for ? Meena : You've got almost everything but you need a pear, the steak and some ginger. The problem with language models as commonsense is that they are collections of patterns and associations, and that…

unlike my dog for example! A more relevant analogy might be a talking parrot :)

Parrots might have what we call a language model... They definitely have inference and autonomy!

Re: Turing-NLG: A 17B-parameter language model

#133

Post by tastroder »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

> Basically, transformer models are the best for NLP.

Yes, this year. While transformers certainly present a breakthrough in the NLP community and certainly stir up the state-of-the-art again, I don't really see how you go from that to the "computers will understand us" conclusion to be honest. People said that during the word2vec stir up and what we got out of that was incremental results (which is not bad, it's in the nature of things really).

We can already build models that do everything you describe as single tasks, while that's exciting, it's not going to lead to the singularity. We've got a long way to go in terms of understanding models, making them computationally tractable, and making them do what we want in the first place without resorting to hoping that our unsupervised model learns something useful. It's likely that the attention mechanisms we see today will be a large part of that but I'm honestly a bit baffled at the "People are vastly underestimating the changes that are about to come from NLP." part. They're not, people already think that today's AI is magic, I don't think it is benefitial to reinforce that. Speech is nuanced, we're making good progress in many areas but we're not on the cusp of any revolutionary change in computational understanding really.

Re: Turing-NLG: A 17B-parameter language model

#134

Post by ragebol »

All these language generation models, in short, base their next word solely on the previous words, right? I'd expect that these generators can be conditioned on e.g. some fact (like in first order logic etc) to express something I want. This is roughly the inverse of for example Natural Language Understanding.

Does anything like this exist?

Re: Turing-NLG: A 17B-parameter language model

#135

Post by timgilbert »

saurkt wrote: post #11

One of the team members from Project Turing. Happy to answer any questions.

I'm honestly not trolling with this question, but can you explain what the practical applications of text generation are? From what I've seen of GPT-2, it's a cool toy, but I have never seen it create anything that seems like it would be useful to solve a problem (eg, a human-computer interaction problem).

The only applications I can think of for text generation are malevolent ones: I'm sure it would be great at generating spam sites which can fool Google's PageRank algorithms, and it seems like you could easily use it in an information warfare / astroturf setting where you could generate the illusion of consensus by arming a lot of bots with short, somewhat convincing opinions about a certain topic.

Is there something obvious I'm missing? It seems too imprecise to actually deliver meaningful information to an end-user, so I'm frankly baffled as to what its purpose is.

Re: Turing-NLG: A 17B-parameter language model

#136

Post by choppaface »

saurkt wrote: post #11

One of the team members from Project Turing. Happy to answer any questions.

“We are releasing a private demo of T-NLG, including its freeform generation, question answering, and summarization capabilities, to a small set of users within the academic community for initial testing and feedback.”

What’s the deal with these private demos? (GPT-2 was also essentially private). More importantly, why even announce the existence of a private demo to people who were not invited?

Re: Turing-NLG: A 17B-parameter language model

#137

Post by gchq-7703 »

ragebol wrote: post #134

All these language generation models, in short, base their next word solely on the previous words, right? I'd expect that these generators can be conditioned on e.g. some fact (like in first order logic etc) to express something I want. This is roughly the inverse of for example Natural Language Understanding. Does anything like this exist?

I'm fairly sure that these models don't work solely on the previous word, but instead are able to remember some level of information from history.

Otherwise, you'd reach a word like 'and' and couldn't possibly follow it with a logical statement that follows on from the previous part.

Re: Turing-NLG: A 17B-parameter language model

#138

Post by ragebol »

gchq-7703 wrote: post #137
ragebol wrote: post #134

All these language generation models, in short, base their next word solely on the previous words, right? I'd expect that these generators can be conditioned on e.g. some fact (like in first order logic etc) to express something I want. This is roughly the inverse of for example Natural Language Understanding. Does anything like this exist?

I'm fairly sure that these models don't work solely on the previous word, but instead are able to remember some level of information from history. Otherwise, you'd reach a word like 'and' and couldn't possibly follow it with a logical statement that follows on from the previous part.

This is why I said 'words', multiple :-).

My point being that these generation models should be conditioned on something more than just word history, like something they want/are instructed to express.

Re: Turing-NLG: A 17B-parameter language model

#140

Post by skocznymroczny »

corporateslave5 wrote: post #3

People are vastly underestimating the changes that are about to come from NLP. The basic ideas of how to get language models working are just about in place. Transformer networks, and recent innovations like GPT-2, googles reformer model, etc are precursors to the real machine learning boom. Machine learning as we have known it, has been stuck as an optimization tool, and used for computer vision here and there. NLP,…

Perhaps, but people overestimate the progress in AI also.

Remember Google's demo of AI reserving a spot at a barber shop? Yeah... that never happened, even though it was supposed to be any day now.

Post reply on HN