Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: A Disappointing Paper?

greaterwrong.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: GPT-3: A Disappointing Paper?

#71

Post by sillysaurusx »

p1esk wrote: post #66
6gvONxR4sf7o wrote: post #36

Earlier quoted context omitted.

The big one is 175 billion parameters. With your hardware's usual 32 bit floats, that's a 700GB model. You won't be using the big one for a while.

This one uses FP16, so you just need to have a server with >350GB of RAM. 512GB of DDR4 would set you back around two grand. A total cost of a server for this would probably be under $5k. Comparable to a good gaming rig.

A TPU can allocate 300GB without OOMing on the TPU's CPU. That's tantalizingly close to 350GB. And 300GB + 8 cores * 8GB = 364GB.

It'll take some work, but I think I can come up with something clever to dump samples on a TPUv2-8. i.e. the free one that comes with Colab.

Realistically, I don't think OpenAI will release the model. Why would they? And I'm not sure they'd dare use "it might be dangerous" as an excuse.

Re: GPT-3: A Disappointing Paper?

#72

Post by sanxiyn »

master_yoda_1 wrote: post #55
ericjang wrote: post #31

I could not disagree more with this post. To summarize what the author is unhappy with: 1) "It’s another big jump in the number, but the underlying architecture hasn’t changed much... it’s pretty annoying and misleading to call it “GPT-3.” GPT-2 was (arguably) a fundamental advance, because it demonstrated the power of way bigger transformers when people didn’t know about that power. Now everyone knows, so it’s the f…

I think we should not compare if anybody is capable or not. Here the most of the work is done by azure engineers and none of them are mention in the paper. So no even open ai can’t do it without azure infrastructure.

OpenAI wrote a new GPU kernel to do this. It's a collaborative work.

Re: GPT-3: A Disappointing Paper?

#73

Post by p1esk »

sillysaurusx wrote: post #71
p1esk wrote: post #66

Earlier quoted context omitted.

This one uses FP16, so you just need to have a server with >350GB of RAM. 512GB of DDR4 would set you back around two grand. A total cost of a server for this would probably be under $5k. Comparable to a good gaming rig.

A TPU can allocate 300GB without OOMing on the TPU's CPU. That's tantalizingly close to 350GB. And 300GB + 8 cores * 8GB = 364GB. It'll take some work , but I think I can come up with something clever to dump samples on a TPUv2-8. i.e. the free one that comes with Colab. Realistically, I don't think OpenAI will release the model. Why would they? And I'm not sure they'd dare use "it might be dangerous" as an excuse.

Have you (or anyone) tried running GPT-2 inference in INT8 precision? Perhaps worth looking at one of these efforts: https://www.google.com/search?q=running+transformer+in+int8&...

Re: GPT-3: A Disappointing Paper?

#74

Post by mattkrause »

p1esk wrote: post #65
freeone3000 wrote: post #34

Earlier quoted context omitted.

The 24GB card mentioned is almost certainly a RTX Titan, which are $3000 each. Just the card. Second, training frameworks like Megatron can distribute to multiple GPUs in the same computer as if they were on different machines, but the naive trainer is greatly helped by NVLink in order to actually look the memory and greatly improve accuracy, which means V100s which are $5000 each. (Also, people use Linux for ML)

An average cost of an NLP researcher is probably around $300k/year. If buying 10 $5k cards makes them twice more productive, then it's a no brainer.

The average cost of an academic NLP researcher is probably closer to $30k/year.

Re: GPT-3: A Disappointing Paper?

#75

Post by p1esk »

The_rationalist wrote: post #69
p1esk wrote: post #67

Earlier quoted context omitted.

I think using AE for text generation is a good idea, and is pretty old one (I tried it myself back in 2015 without particularly good results), but I wouldn't call it a breakthrough. To me a breakthrough/new paradigm would be something like this: https://arxiv.org/abs/1906.05317

The breakthrough is not that they are the first to use AE for text. It is that they are the first to eliminate the bottleneck that prevented AEs to beat transformers. You linked paper seems interesting on paper but does it bring any new SOTA?

AEs to beat transformers

Can you point me to some example of generated text this model produced? Something similar in quality to that unicorn story from GPT-2?

but does it bring any new SOTA?

Looking at their tables, seems so. The code is open source, and there's demo at https://mosaickg.apps.allenai.org/

Re: GPT-3: A Disappointing Paper?

#76

Post by p1esk »

mattkrause wrote: post #74
p1esk wrote: post #65

Earlier quoted context omitted.

An average cost of an NLP researcher is probably around $300k/year. If buying 10 $5k cards makes them twice more productive, then it's a no brainer.

The average cost of an academic NLP researcher is probably closer to $30k/year.

Most universities have access to supercomputers, including GPU clusters. But that's not the point, not every NLP problem requires experimenting with 175B parameter models.

Academic researchers shouldn't try to compete with Google or OpenAI in scaling up models. They should try to come up with new approaches. Our brains have been evolving under tight constraints (size, energy, noise, etc). Maybe a good academic problem to solve is "how can I do what GPT-3 does if I only have an 8 GPU workstation?" This might lead to all kinds of breakthroughs.

Re: GPT-3: A Disappointing Paper?

#77

Post by [deleted] »

clickok wrote: post #53
KKKKkkkk1 wrote: post #45

Earlier quoted context omitted.

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his r…

You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of…

[deleted]

Re: GPT-3: A Disappointing Paper?

#78

Post by KKKKkkkk1 »

clickok wrote: post #53
KKKKkkkk1 wrote: post #45

Earlier quoted context omitted.

I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his r…

You have misunderstood Sutton's argument. Quoting Rich: One thing that should be learned from the bitter lesson is the great power of general purpose methods, of methods that continue to scale with increased computation even as the available computation becomes very great. The point isn't that improvements in our algorithms is unnecessary or unhelpful, rather that the algorithms we should focus on will be capable of…

I think this argument is neither here nor there. For computer vision problems, we use convnets, which are models inspired by a biological model of vision. By doing that we are embedding our preconceived notions of what vision is into our models instead of throwing compute and data at the problem. Earlier attempts using multi-layer perceptrons have been massive failures. Is this consistent with Sutton's analysis or contrary to it?

Re: GPT-3: A Disappointing Paper?

#79

Post by 6gvONxR4sf7o »

p1esk wrote: post #66
6gvONxR4sf7o wrote: post #36

Earlier quoted context omitted.

The big one is 175 billion parameters. With your hardware's usual 32 bit floats, that's a 700GB model. You won't be using the big one for a while.

This one uses FP16, so you just need to have a server with >350GB of RAM. 512GB of DDR4 would set you back around two grand. A total cost of a server for this would probably be under $5k. Comparable to a good gaming rig.

It uses FP16 yes, but the question was about average people running them on their PCs. I don't think most PCs have fp16 support, so you'd have to do it in fp32, doubling the size. It's likely not so fast on a CPU either with that size, especially when using FP32.

Re: GPT-3: A Disappointing Paper?

#80

Post by remexre »

jkhdigital wrote: post #63

> it would represent a kind of non-linguistic general intelligence ability which would be remarkable to find in a language model As a relative outsider to this field, I don’t really see the stark line between natural language and general intelligence implied by this statement. Language is just abstractions encoded in symbols, and general intelligence is just the ability to construct and manipulate abstractions. Seems…

I think the line you'd see is that there exists some task where the language-based model suddenly lacks the ability to perform the task despite the fact that it "should."

I'd conjecture that this might include something like describing where places are in relation to each other, and asking it to describe a route. (Not an NLP expert, but work with AI folks; this task chosen as an example because it seems like something you'd want a planner for rather than anything MLful.)

Post reply on HN