Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-3: Language Models Are Few-Shot Learners

arxiv.org

Post reply on HN
Print view View original

171–180 of 212 posts

Re: GPT-3: Language Models Are Few-Shot Learners

#171

Post by 6gvONxR4sf7o »

In the paper they say it took 3.14e23 flops to train. They used v100s to do it. This is an insane energy cost (and financial cost).

Nvidia's v100 product page [0] says that it gets about 15 (single precision) - 125 ("deep learning") teraflop/s at 250-300 watts (joules per second). That means that if everything's as perfectly efficient as a marketing product page, it gets about 250/125-300/15 = 2-25 joules per teraflop, putting this model at about 0.6-8 terajoules.

A gallon of gasoline has about 120e6 joules [1] (though if you wanted to compare with burning it in a car, it's only 20-25% efficient at best [2] so it'd be fewer joules/gallon).

This model took the equivalent of about 5,000-67,000 gallons of gasoline at best and at ideal perfect energy efficiency. I get that openAI has made a decision not to be efficient with their dollars in order to see what's possible with future tech, but that means not being efficient with energy either, and it's getting kinda crazy. Sure, microsoft data centers aren't gasoline powered, so maybe it is closer to this ideal energy efficiency, and it's definitely going to be a better carbon footprint, but god damn it just seems wasteful.

Hell, the new A100 (again going off marketing materials [3], so at least it's apples to apples) could do it about 4x more efficiently. Is this research really worth what it costs, when waiting a year makes it that much more efficient?

[0] https://www.nvidia.com/en-us/data-center/v100/

[1] https://www.calculateme.com/energy/gallons-of-gas/to-joules/....

[2] https://en.wikipedia.org/wiki/Engine_efficiency#Gasoline_(pe...

[3] https://devblogs.nvidia.com/nvidia-ampere-architecture-in-de...

Re: GPT-3: Language Models Are Few-Shot Learners

#172

Post by Imnimo »

gambler wrote: post #164
cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

I'll wait for a working interactive model before blindly believing these statements. GPT-2 was hyped through the roof, but when inspected with a bit of criticality it demonstrated glitches that told us more about how it actually works than "good" examples: https://medium.com/@VictorBanev/interrogating-full-gpt-2-10a... ML models should be pushed to their limit, because that's where you gather most useful information…

The last AI paper I read that has a list of things the model failed at is this one:

https://news.ycombinator.com/item?id=23345379

See Section 5, titled "Limitations"

Re: GPT-3: Language Models Are Few-Shot Learners

#173

Post by cs702 »

skdotdan wrote: post #147
cs702 wrote: post #130

This looks like a big deal to me: 1. First of all, the authors successfully trained a model with 173 BILLION PARAMETERS. The previous largest model in the literature, Google’s T5, had "only" 11 billion. With Float32 representations, GPT-3-173B's weights alone occupy ~700GB of memory (173 billion params × 4 bytes/param). A figure in the 100's of billions is still 3 orders of magnitude smaller than the 100’s of trillio…

AFAIK they used half-precision (Float16)

Thanks. I should have written "if using Float32," which is what I meant -- instead of "with Float32," which in hindsight reads a bit ambiguous. But regardless of which floating-point representation is used, the number of weights is still in the hundreds of billions... which is insane.

Re: GPT-3: Language Models Are Few-Shot Learners

#175

Post by saulrh »

6gvONxR4sf7o wrote: post #171

In the paper they say it took 3.14e23 flops to train. They used v100s to do it. This is an insane energy cost (and financial cost). Nvidia's v100 product page [0] says that it gets about 15 (single precision) - 125 ("deep learning") teraflop/s at 250-300 watts (joules per second). That means that if everything's as perfectly efficient as a marketing product page, it gets about 250/125-300/15 = 2-25 joules per teraflo…

What's the TCO of a few hundred teenagers? I haven't read the paper yet, but if the other comments here are accurate, that's about what you'd have to shell out for if you wanted to duplicate the productivity of this mdoel without externalizing costs by e.g. offering unpaid internships to high school students.

Re: GPT-3: Language Models Are Few-Shot Learners

#176

Post by gambler »

gwern wrote: post #169
gambler wrote: post #164

Earlier quoted context omitted.

I'll wait for a working interactive model before blindly believing these statements. GPT-2 was hyped through the roof, but when inspected with a bit of criticality it demonstrated glitches that told us more about how it actually works than "good" examples: https://medium.com/@VictorBanev/interrogating-full-gpt-2-10a... ML models should be pushed to their limit, because that's where you gather most useful information…

That's a very sloppy post. He does a single example, not even running locally or changing sampling parameters, and then concludes that GPT-2 is doing nothing but pattern-matching? A lot of people underestimate NNs because the sampling from them (top-k! how much dumber and cruder can you get? nucleus works better, but is still obviously suboptimal) destroys a lot of dark knowledge. I noticed this with Gary Marcus's cl…

The article I linked to makes claims only about the model it tests and since it actually links to an online implementation, anyone can try to reproduce the results and see for themselves. This is more than I can say about most chatter about ML.

>Such tests can prove the presence of knowledge, but not the absence...

This sounds like a setup for non-falsifiable beliefs.

Re: GPT-3: Language Models Are Few-Shot Learners

#177

Post by 6gvONxR4sf7o »

saulrh wrote: post #175
6gvONxR4sf7o wrote: post #171

In the paper they say it took 3.14e23 flops to train. They used v100s to do it. This is an insane energy cost (and financial cost). Nvidia's v100 product page [0] says that it gets about 15 (single precision) - 125 ("deep learning") teraflop/s at 250-300 watts (joules per second). That means that if everything's as perfectly efficient as a marketing product page, it gets about 250/125-300/15 = 2-25 joules per teraflo…

What's the TCO of a few hundred teenagers? I haven't read the paper yet, but if the other comments here are accurate, that's about what you'd have to shell out for if you wanted to duplicate the productivity of this mdoel without externalizing costs by e.g. offering unpaid internships to high school students.

GPT-2 came out about a year and a quarter ago. GPT came out less than a year before. If we take another commenter's estimate of $3.6M, and a new model comes out every year or so, then you could say just training is like $3.6M per year. That should cover a pretty large number of teenagers. Hell, that would cover a whole early stage startup in san francisco, including office space.

Re: GPT-3: Language Models Are Few-Shot Learners

#178

Post by Yajirobe »

6gvONxR4sf7o wrote: post #171

In the paper they say it took 3.14e23 flops to train. They used v100s to do it. This is an insane energy cost (and financial cost). Nvidia's v100 product page [0] says that it gets about 15 (single precision) - 125 ("deep learning") teraflop/s at 250-300 watts (joules per second). That means that if everything's as perfectly efficient as a marketing product page, it gets about 250/125-300/15 = 2-25 joules per teraflo…

That sucks. Future SOTA AI models are going to be completely out of reach for hobbyists.

Re: GPT-3: Language Models Are Few-Shot Learners

#179

Post by KhoomeiK »

MauranKilom wrote: post #140
ragebol wrote: post #110

Earlier quoted context omitted.

My thoughts exactly. It seems a person incapable of proper grammar (like a baby) has some concept or thought it wants to express, but can't because it doesn't know the words etc. These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express. There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-…

I have been thinking about this kind of thing too. What if there was some way to feed your condensed thoughts into such a model and it writes a paper/blog post/article? Essentially, one should be able to use these models to "interpolate" the writing around the raw meaning/content. Typing assistance (think Grammarly) already allows you to refine finished writing to be more in line with what some language model expects…

What might some valid sources of data for this be? Perhaps comparisons of Simple English Wikipedia to the standard English Wikipedia? We'd need a side-by-side comparison of condensed information and a fluffed up piece.

Re: GPT-3: Language Models Are Few-Shot Learners

#180

Post by h3ctic »

zwaps wrote: post #92
sytelus wrote: post #88

GPT-3/175B model required 3.14E23 flops of compute for training. Even at theoretical 28 TFLOPS for V100 and lowest reserved Azure pricing, this will take 355 GPU-years and cost $3.6M for a single training run!

I know it's a meme for the GPT team to just take the latest transformer model and add a magnitude of parameters, done and done! It'll be interesting to see whether the new paradigm really offers new insights, or whether it's really just kicking the can down the road - and we see the limits of generalizability in some other fashion. I guess what irks me is that there is so little theory and math behind many papers, ev…

Can you point out some books/authors/papers to close the gap between statistics and NNs?
Post reply on HN