Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: A Disappointing Paper?

greaterwrong.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: GPT-3: A Disappointing Paper?

#61

Post by cs702 »

pacala wrote: post #59
cs702 wrote: post #13

Earlier quoted context omitted.

GPT-3 was pretrained on five datasets (Common Crawl, WebText2, Books1, Books2, and Wikipedia; see table 2.2), and then used on previously unseen tasks (Q&A, translation, cloze, etc.) without finetuning , i.e., weights were not updated after the original (autoregressive) pretraining. This promises a possible future in which general-purpose models are pretrained once, and deployed to production for multiple tasks.

This is pretty mind-boggling, to the point of suspecting an error in the methodology. If it is just a language model, then it has no baked in notion of test time tasks. How on earth does a language model know what is required of it at test-time without fine tuning? How does it know that the test time prompt are examples of the task, and not some story prompt it's supposed to riff off in random ways?

Yes, mind-boggling.

For each task, the authors feed a context window of text with either zero to a few sample queries and responses, followed by a query without the response. The model generates a response for the last query. BTW, this approach is analogous to what you would do with a human being: you would provide zero to a few sample questions and answers, and then ask a question.

Re: GPT-3: A Disappointing Paper?

#62

Post by [deleted] »

cs702 wrote: post #61
pacala wrote: post #59

Earlier quoted context omitted.

This is pretty mind-boggling, to the point of suspecting an error in the methodology. If it is just a language model, then it has no baked in notion of test time tasks. How on earth does a language model know what is required of it at test-time without fine tuning? How does it know that the test time prompt are examples of the task, and not some story prompt it's supposed to riff off in random ways?

Yes, mind-boggling. For each task, the authors feed a context window of text with either zero to a few sample queries and responses, followed by a query without the response. The model generates a response for the last query. BTW, this approach is analogous to what you would do with a human being: you would provide zero to a few sample questions and answers, and then ask a question.

[deleted]

Re: GPT-3: A Disappointing Paper?

#63

Post by jkhdigital »

> it would represent a kind of non-linguistic general intelligence ability which would be remarkable to find in a language model

As a relative outsider to this field, I don’t really see the stark line between natural language and general intelligence implied by this statement. Language is just abstractions encoded in symbols, and general intelligence is just the ability to construct and manipulate abstractions. Seems reasonable to think that these are two sides of the same coin.

Put another way, natural language is the product of general intelligence.

Re: GPT-3: A Disappointing Paper?

#64

Post by canjobear »

The_rationalist wrote: post #2

Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2

The perplexity numbers are for different tasks. MIM is encoding a sentence into a latent variable and then reconstructing it, and achieves PTB perplexity 4.6. GPT-2 is generating the sentence from scratch, which will on average have higher perplexity numbers. I agree that PTB perplexity 4.6 on autoregressive language modeling would be a huge result.

Re: GPT-3: A Disappointing Paper?

#65

Post by p1esk »

freeone3000 wrote: post #34
andi999 wrote: post #27

Earlier quoted context omitted.

I am doing other hpc stuff, so I am wondering why are you limited to 1 GPU? Windows recognizes up to 8 I think.

The 24GB card mentioned is almost certainly a RTX Titan, which are $3000 each. Just the card. Second, training frameworks like Megatron can distribute to multiple GPUs in the same computer as if they were on different machines, but the naive trainer is greatly helped by NVLink in order to actually look the memory and greatly improve accuracy, which means V100s which are $5000 each. (Also, people use Linux for ML)

An average cost of an NLP researcher is probably around $300k/year. If buying 10 $5k cards makes them twice more productive, then it's a no brainer.

Re: GPT-3: A Disappointing Paper?

#66

Post by p1esk »

6gvONxR4sf7o wrote: post #36
drcode wrote: post #28

Newbie question: If/when models the size of GPT3 are released to the general public, will average people going to be able to run them on their PCs, as they can with GPT2? Or will that basically be impossible now without expensive specialty hardware?

The big one is 175 billion parameters. With your hardware's usual 32 bit floats, that's a 700GB model. You won't be using the big one for a while.

This one uses FP16, so you just need to have a server with >350GB of RAM. 512GB of DDR4 would set you back around two grand. A total cost of a server for this would probably be under $5k. Comparable to a good gaming rig.

Re: GPT-3: A Disappointing Paper?

#67

Post by p1esk »

The_rationalist wrote: post #2

Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2

I think using AE for text generation is a good idea, and is pretty old one (I tried it myself back in 2015 without particularly good results), but I wouldn't call it a breakthrough. To me a breakthrough/new paradigm would be something like this: https://arxiv.org/abs/1906.05317

Re: GPT-3: A Disappointing Paper?

#68

Post by The_rationalist »

phreeza wrote: post #60
The_rationalist wrote: post #6

Earlier quoted context omitted.

why you feel this paper so important? the best summary are the benchmarks: first place at question answering (on yahoo task) First place on language modeling for pen treebank (by FAR) So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year. It achiev…

Not all models scale up well with more parameters. VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015. https://arxiv.org/abs/1511.06349

What's new is that they are the first to eliminate the bottleneck that prevented VAES to beat transformers

Re: GPT-3: A Disappointing Paper?

#69

Post by The_rationalist »

p1esk wrote: post #67
The_rationalist wrote: post #2

Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2

I think using AE for text generation is a good idea, and is pretty old one (I tried it myself back in 2015 without particularly good results), but I wouldn't call it a breakthrough. To me a breakthrough/new paradigm would be something like this: https://arxiv.org/abs/1906.05317

The breakthrough is not that they are the first to use AE for text. It is that they are the first to eliminate the bottleneck that prevented AEs to beat transformers.

You linked paper seems interesting on paper but does it bring any new SOTA?

Re: GPT-3: A Disappointing Paper?

#70

Post by The_rationalist »

Eridrus wrote: post #19
cs702 wrote: post #7

All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…

I'm an engineer and this is a boring paper. They didn't beat the conclusively beat SOTA the way BERT/T5 beat basically everything. Sure, this is "unsupervised", but from an engineering perspective that's entirely uninteresting, so we don't even get actual scaling numbers. This paper was written to pretend it is more than it is, which is basically the hallmark of everything OpenAI does.

There is one model that consistently outperform BERT: it's XLnet but it's already one year old, innovation seems to slow down
Post reply on HN