Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Better Language Models and Their Implications

blog.openai.com

Post reply on HN
Print view View original

41–50 of 138 posts

Re: Better Language Models and Their Implications

#41

Post by levesque »

fouc wrote: post #36

I think this would be extremely useful when we can do the inverse. Basically - can we detect if someone's writing is nonsensical or not? Can we detect if someone that is producing many well written essays is adhering to reality or not? Are they subtly re-defining terms, using flawed examples, etc? The generated example of the biologists discovering a unicorn herd is too convincing on its own. It's only because it's s…

Our models have reviewed your submission, and deemed it to be 87% incoherent, 52% redundant, and 24% fake. We therefore reject your submission. Sincerely, the Chief Bot Editor.

Re: Better Language Models and Their Implications

#42

Post by lucidrains »

exit wrote: post #34
lucidrains wrote: post #22

Started a Google colab with the interactive text generation script. https://colab.research.google.com/drive/1da54684tFMjPbR5idbv...

to be clear, this is the "politically innocuous" open sourced model. the results are not impressive.

Yea, somewhere in between the released model and the model they presented, the deep net has learned narrative structure.

Re: Better Language Models and Their Implications

#43

Post by Franciscouzo »

aaaaaaaaaaab wrote: post #40

In 10 years, content written by actual humans will be a premium niche, like tailored suits - reserved for the elites. The rest of us will be force-fed with machine-generated garbage.

That doesn't makes sense, text can be distributed at marginal cost, tailored suits are expensive because there's not a lot of supply.

Re: Better Language Models and Their Implications

#44

Post by Nydhal »

levesque wrote: post #38
avivo wrote: post #31

This was only a matter of time. For the DEFCON AI Village in August I talked about the implications of this sort of tech, and how that impacts how we release "exploit" code / think about "cognitive vulnerabilities": https://medium.com/@aviv/what-does-a-world-with-automated-so... . If you are doing work in this space, either in ML research or related security, you need to be thinking about implications (also see e.g.…

I mean, the ideas are there. The scope of the project is probably too big to reproduce for now, but eventually it will be accessible to your average spammer / scammer. We will get there. We won't be able to get these tools locked, make them exclusive for a certain type of responsible AI specialists. Someone will spill the beans, the models. People with bad intentions will reproduce these results. To me, the real deal…

On the flip side, this will give an edge to specific groups that can communicate in highly vernacular/non-written/non-standard languages.

Re: Better Language Models and Their Implications

#45

Post by cs702 »

This kind of "blocking-and-tackling" work is important.

The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks.

Remarkably, the performance of a Transformer in the tested tasks improves log-linearly with the number of parameters, suggesting that even the largest model tested, with 1.5B parameters, still underfits 40GB of text.

This is compelling evidence that we do NOT need new architectures, NOR new kinds of training objectives, NOR new theories, for better language modeling! We can get better language modeling simply by increasing model capacity (i.e., by adding more parameters to existing models), which becomes easier and simpler to do as hardware continues to improve over time.

Great work.

PS. In case it's not clear: I'm not saying we should suddenly stop searching for new, better ideas and architectures. That would be silly. Please don't attack a straw-man :-)

[a] https://arxiv.org/abs/1706.03762

Re: Better Language Models and Their Implications

#46

Post by aaaaaaaaaaab »

Franciscouzo wrote: post #43
aaaaaaaaaaab wrote: post #40

In 10 years, content written by actual humans will be a premium niche, like tailored suits - reserved for the elites. The rest of us will be force-fed with machine-generated garbage.

That doesn't makes sense, text can be distributed at marginal cost, tailored suits are expensive because there's not a lot of supply.

Tailored suits used to be cheap too, but taylors went out of business due to mass production.

Re: Better Language Models and Their Implications

#48

Post by modeless »

cs702 wrote: post #45

This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performanc…

Agreed that "simply" scaling up with more compute will result in progress and useful systems, and work in that direction is interesting and valuable. But, while we may not need new architectures or training objectives to make progress, we do need them to approach human level sample complexity. Humans don't need to read through 40 GB of text multiple times to learn to write.

Re: Better Language Models and Their Implications

#49

Post by fartcannon »

cs702 wrote: post #45

This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performanc…

I feel like my human brain underfits a lot of data.

My human brain definite zero-shot transfer learns almost everything.

Re: Better Language Models and Their Implications

#50

Post by rosstaylor90 »

cs702 wrote: post #45

This kind of "blocking-and-tackling" work is important. The authors take a well-known architecture, the Transformer[a], configure it with a progressively larger number of parameter, train it to predict the next word conditioned on previous text, using a large dataset consisting of 40GB of text scraped from the Web, and test each trained model on a range of zero-shot transfer-learning tasks. Remarkably, the performanc…

I think your conclusion is too strong. Yes, we know bigger models and data generally lead to better performance (e.g. BigGAN results last year) but progress in architecture can still speed up progress in ML tasks. If we were still stuck using RNNs and LSTMs for language modeling we wouldn't be talking about this news today.
Post reply on HN