Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Language Models and Their Implications

blog.openai.com

Post reply on HN
Print view View original

81–90 of 138 posts

Re: Better Language Models and Their Implications

#81

Post by Cybiote »

cs702 wrote: post #54
modeless wrote: post #48

Earlier quoted context omitted.

Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Humans don't need to read through 40 GB of text multiple times to learn to write.

> Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Yes, agreed. Nothing I said above contradicts that! :-) > Humans don't need to read through 40 GB of text multiple times to…

Conscious processes are estimated to work on the order of 10^2 bits. Vision, at the retina, is estimated at 10^7 bits/sec. It drops another order of magnitude by V1. Also note that long as they're not isolated, a deaf and blind person has no trouble getting to full human reasoning ability despite being vastly more impoverished in available data compared to the average person.

A human will also be learning vision, hearing, walking, physics, causal reasoning, and much more. This comparison just isn't well grounded. Task specific is how much training does a young brain require to learn to produce language? If the brain comes with innate advantages then rather than be resort to inefficiency and excusing our models, we should try to see if they can be bettered.

Re: Better Language Models and Their Implications

#82

Post by DenisM »

minimaxir wrote: post #12

The generated text sounds too good; is it possible that the model overfit the source material (especially since the n-previous-tokens value is infinite, while other approaches like char-rnns/textgenrnn use a fixed window length)? It's something I've encountered many times while working with text generation.

It is uncanny.

I would expect a language model to pull up different person names for each time that one was called for. For a person name to be consistently used through several paragraphs it is not enough to rely on word co-occurrence.

If I had to produce a text like this I would simply take an existing text and replace randomly chosen words with other similar words (as hinted by amvalo). Similar as in - words that tend to occur in similar contexts. So John->Bob throughout entire text. But that would not be a language model product anymore, and where is fun in that?

I should set aside some time to read this paper.

Re: Better Language Models and Their Implications

#84

Post by ninjin »

cs702 wrote: post #76
ninjin wrote: post #67

Earlier quoted context omitted.

> …we do NOT need new architectures… You do realise the irony in stating this regarding transformers that arguably made their first appearance as decomposable attention in 2016 [1] and then as transformers in 2017 [2]? It is not as if this is a vanilla RNN straight out of the 90s sweeping the floor with decades of model innovations, rather it looks like we are seeing the rise of a new, simple model category that work…

The point I was trying to make is that we can get better language models with more computation, not that we should stop researching new ideas and architectures. In hindsight, perhaps the language in my post wasn't sufficiently clear on this. See my PS. You and I are not in disagreement :-)

I do not get it. If what you are trying to say is: “more data/computation helps”, then why even bother throwing in models, training objectives, and theory into your statement? We have known that more data helps for at least two decades, so saying this is stating the obvious. Are you then trying to say: “Right at this very moment we can push a bit further with more data/computation given what we have for this specific task”, well, then again this is nothing particularly insightful to bring to the table as surely this occurs at least once a month for a given task/model category.

This is not a matter of clarity, it is a matter of form – drop the hyperbole and speak specifics rather than trying to appease both sides of a fictitious divide.

Re: Better Language Models and Their Implications

#85

Post by wuthefwasthat »

Cybiote wrote: post #51

It's becoming ever more certain that the transformer architecture is one of the largest contributions to AI (not merely machine learning, but AI), often beating LSTMs despite LSTMs being expressive enough to capture Turing Equivalence (at least in theory). Its main ideas are three: shorter paths help gradient flow, the training setup and the final key aspect, unhelpfully called self-attention. Self-attention is bette…

Yeah, there are definitely still places the samples fall short! Keep in mind we're still using very naive sampling techniques.

RE Winograd: WNLI is different, see https://arxiv.org/pdf/1804.07461.pdf

Re: Better Language Models and Their Implications

#86

Post by bitL »

aaaaaaaaaaab wrote: post #40

In 10 years, content written by actual humans will be a premium niche, like tailored suits - reserved for the elites. The rest of us will be force-fed with machine-generated garbage.

But will ML companies pay link tax? Assuming ML would write/fill in forms to incorporate itself, have a conversation or two with some tax office person, and then off to the wild...

Where can I study computational law to stay ahead of the curve?

Re: Better Language Models and Their Implications

#88

Post by olleromam91 »

dzink wrote: post #25

This tech can easily be used to flood humanity’s shared brain with auto-generated propaganda. Schizophrenia of the internet in a way. There is plenty of incentive with Google algorithms favoring number of words and relevant keywords in content for rankings - you could have NLP bots lifting junk sites to top results. To step ahead in that chess game, a detection tool for fake would be just training grounds for better…

Great Idea...but better make sure it's not hosted on the internet.

Re: Better Language Models and Their Implications

#90

Post by xiphias2 »

schoen wrote: post #73
wuthefwasthat wrote: post #3

Earlier quoted context omitted.

Thank you! We've released 500 random unconditional samples from GPT-2 at https://github.com/openai/gpt-2/blob/master/gpt2-samples.txt

Wow, some of these really go off the rails but those that only kinda go off the rails are absolutely hilarious and/or bizarre. A few summaries of ones that I looked at which appeared to be more or less staying on a single topic: Sample 1: An Austin nonvegetarian vegetarian restaurant encounters a series of difficulties in opening, as its nonexistent but extensive menu depicts a wide range of food options and the rest…

I'm looking forward to computers generating short clips from these stories. I wouldn't be surprised if some of them get viral (especially if A/B testing would be incorporated by looking at when the users stop watching the videos).
Post reply on HN