GPT-3/175B model required 3.14E23 flops of compute for training. Even at theoretical 28 TFLOPS for V100 and lowest reserved Azure pricing, this will take 355 GPU-years and cost $3.6M for a single training run!
[deleted]
GPT-3: Language Models Are Few-Shot Learners
161–170 of 212 posts
Re: GPT-3: Language Models Are Few-Shot Learners
#162Earlier quoted context omitted.
Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…
Very interesting comment, thanks for taking the time to write it :) I think if memory is the only problem than optimizing training time should be more of a concern. I'm imagining a huge language model than can retrain very quickly. So I suppose it might be a decent idea to not measure it by perplexity or some human judgement score or whatever but rather by that score per compute units used. Or in other words...maybe…
> I like the idea of one "base model" that represents language and many different context models on top of it (finetuning the core model)
This is an entirely different concept of computer language than the current GPT style models. These systems don't "represent language", and cannot. The whole reason why GPT is so exciting right now is that it fundamentally threw away the entire concept of "representing language". That has some upsides ... and some downsides.
Re: GPT-3: Language Models Are Few-Shot Learners
#163Earlier quoted context omitted.
Dude, I’m sorry, but the average person will not know the difference between that and a regular buzzfeed article or YouTube comment. We’re not going to need ad blockers in the future, we won’t even need these visual ads on websites anymore. There will be trained bots that can promote any idea/product and pollute comments and articles. It’s over, we lost. Morpheus: What if I told you that, throughout your whole life,…
Hello. Gwern and I trained the GPT-2 1.5B model that powers /r/SubSimulatorGPT2. https://www.reddit.com/r/SubSimulatorGPT2/ I've been basically living and breathing GPT-2 for ... gosh, it's been 6 months or so. The past few months have been a lot of StyleGAN2 and a lot of BigGAN, but before that, it was very "make GPT-2 sing and dance in unexpectedly interesting ways" type work. I don't claim to know a lot. But occas…
Thanks for this comment, and thanks for all your work on generative models.
Re: GPT-3: Language Models Are Few-Shot Learners
#164This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…
I'll wait for a working interactive model before blindly believing these statements. GPT-2 was hyped through the roof, but when inspected with a bit of criticality it demonstrated glitches that told us more about how it actually works than "good" examples:
https://medium.com/@VictorBanev/interrogating-full-gpt-2-10a...
ML models should be pushed to their limit, because that's where you gather most useful information about what they actually do. Their results need to be critically examined with both exploratory and hypothesis-driven testing. And yet this is never done in initial papers and rarely done afterwards.
What was the last AI paper you've read that that said "and here is a list of things out model failed at"?
Re: GPT-3: Language Models Are Few-Shot Learners
#165Earlier quoted context omitted.
In Sam A's words, "genuinely, we have an algorithm that can learn."
Do you have a source? I am genuinely curious as I can't find it and would like to see the context
https://www.youtube.com/watch?v=0TRtSk-ufu0
Around 19:10~. Though I messed up, he didn't say 'genuinely'. He said "full stop, truly, legitimately, we have an algorithm that can learn".
Re: GPT-3: Language Models Are Few-Shot Learners
#166This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…
> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.
It's not clear whether a parameter in a neural network maps cleanly onto a synapse in a biological brain.
Re: GPT-3: Language Models Are Few-Shot Learners
#167Earlier quoted context omitted.
> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.
It's not clear whether a parameter in a neural network maps cleanly onto a synapse in a biological brain.
I think it's becoming pretty clear that they don't. First, scientists uncovered many additional ways neurons interact with one another[1]. Second, it seems that individual neurons do way more computing than in the simplistic ANN models [2].
Re: GPT-3: Language Models Are Few-Shot Learners
#168I'm kind of scared to see what GPT-10 will be capable of.
But I am really excited to get to play it, to test it out, and to try out my toolset to make sure it will do what I need it to do.
Source: https://talktotransformer.com/ Input: "I'm kind of scared to see what GPT-10 will be capable of."
Re: GPT-3: Language Models Are Few-Shot Learners
#169This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…
I'll wait for a working interactive model before blindly believing these statements. GPT-2 was hyped through the roof, but when inspected with a bit of criticality it demonstrated glitches that told us more about how it actually works than "good" examples: https://medium.com/@VictorBanev/interrogating-full-gpt-2-10a... ML models should be pushed to their limit, because that's where you gather most useful information…
That's a very sloppy post. He does a single example, not even running locally or changing sampling parameters, and then concludes that GPT-2 is doing nothing but pattern-matching? A lot of people underestimate NNs because the sampling from them (top-k! how much dumber and cruder can you get? nucleus works better, but is still obviously suboptimal) destroys a lot of dark knowledge. I noticed this with Gary Marcus's claims about GPT-2 too: he would try once, without changing any sampling settings, and conclude that it wasn't doing anything, but if you tried, you would get different results. I'm not the only one to notice that: https://www.quantamagazine.org/common-sense-comes-to-compute... Such tests can prove the presence of knowledge, but not the absence... And of course, GPT-3 does extensive arithmetic tricks: https://arxiv.org/pdf/2005.14165.pdf#page=22
Re: GPT-3: Language Models Are Few-Shot Learners
#170Earlier quoted context omitted.
> still 3 orders of magnitude smaller than the 100’s of trillions of synapses in the human brain Wow, that is WAY closer than I thought we were.
It's not clear whether a parameter in a neural network maps cleanly onto a synapse in a biological brain.
It’s very clear that they don’t.