Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Democratization in the Era of GPT-3

thegradient.pub

Post reply on HN
Print view View original

41–50 of 90 posts

Re: AI Democratization in the Era of GPT-3

#41

Post by exged »

anoncareer0212 wrote: post #33
mensetmanusman wrote: post #24

I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level with…

Given the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264

That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.

Re: AI Democratization in the Era of GPT-3

#42

Post by decasteve »

jackcosgrove wrote: post #39

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of traini…

The airplane and the radio brought people closer. It also improved the coordination and delivery of mass destruction.

Technology can be a rollercoaster ride of benefit and detriment.

Re: AI Democratization in the Era of GPT-3

#43

Post by lopmotr »

exged wrote: post #41
anoncareer0212 wrote: post #33

Earlier quoted context omitted.

Given the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264

That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.

Is that unit PetaFLOPS/day right? I think it should be PetaFLOPS-day which has dimensions of FLOP (total number of operations), rather than FLOP/time^2.

The cost wouldn't be the cost of the hardware because it still exists afterwards. You'd have to discount it for the amount of time it was in use.

Re: AI Democratization in the Era of GPT-3

#44

Post by liuliu »

ma2rten wrote: post #40
liuliu wrote: post #19

Earlier quoted context omitted.

You need tiny bit of memory for activations if you don't want fine-tuning. I think for GPT-3, fine-tuning is out of window. But it is reasonable to expect inference takes less than a minute with single 3090 and fast enough SSD.

OpenAI offers a fine-tuning api. How did come up the one minute estimate? According to a quick google search I did, the fastest SSDs these days have a bandwidth of 3100 MB/s. So it would take 112s just to read the weights.

I don't have access to see whether they have fine-tuning API. Do you have any links explain the said fine-tuning? It is certainly surprising given there is no fine-tuning experiment mentioned in the GPT-3 paper.

Weights loading is embarrassingly simple to parallelize. Just use madam with 3 or 4 NVMe SSD sticks are sufficiently enough. You are more likely bounded by PCIe bandwidth than the SSD bandwidth. Newer NVIDIA cards with PCIe-4 support helps.

Re: AI Democratization in the Era of GPT-3

#46

Post by elpin »

csande17 wrote: post #36
hanoz wrote: post #22

Earlier quoted context omitted.

> We are generally prioritizing people with a specific application they'd like to build. Why?

OpenAI's goals are (1) make money and (2) generate positive press coverage about OpenAI. (They make statements about wanting other things but that's mainly to help them achieve (2).) Prioritizing people with concrete project ideas helps them in both areas: they're more likely to convert into paid customers down the line, and they're more likely to generate "OpenAI technology is now being used for X" press releases.

I think there's a fair argument that groups attempting to make a specific product are more likely to drive platform development than random individuals who just want to noodle around. This isn't to say that the more individual experimenters won't drive development too, just that when you're dealing with limited resources you do have to make some decisions about allocation.

Just framing it in terms of money and "generating positive press coverage" is a little cynical IMO. Is prioritizing any cool use cases of their technology that push the boundaries of today's technology to create real use cases besides "haha look I can make GPT3 parody VC Medium/LinkedIn articles" just press optics? I don't think so but can also understand the concern especially given this article is about democratization.

Re: AI Democratization in the Era of GPT-3

#47

Post by boulos »

robmsmt wrote: post #38
boulos wrote: post #9

Disclosure: I work on Google Cloud and have worked with the OpenAI folks on large models. This article mixes both “should research be open” and “is this work cheaply reproduced / accessible”: For smaller, open models: > The average person could not recreate models of this size from scratch, but the models can run on a single machine with a single GPU. but about GPT-3: > GPT-3 represents a new circumstance. For the fi…

how big do the models get? are you able to say?

As big as you want, kind of. The challenge, as in large-scale physics, is how many nodes you can stick together with sufficiently high bandwidth (low latency is less important in the ML space, because there are lots of ops per byte, unlike some CFD Simulations that have very few per update).

On our Cloud TPU product page [1], we have a single TPU v3 pod with 32 TB of memory. For the most recent MLperf submission, the TPU folks hooked up four of them [2]. There’s obviously a reduction in scalability from doing so (see weak scaling versus strong scaling terminology), but that’s the interesting co-design question: what kind of models can you usefully train in an “even more distributed” mode?

Outside of TPUs though, even our single 16x A100 offering has 640 GB all connected by NVLINK (other providers went with 8, so 320 GB of “system memory”) and there are at least a few in a single rack. So the era of TiB scale models is certainly “semi feasible” and “open to all”.

The challenge is that you need to also train these for quite some time. 1000 V100s would cost you at least $2000/hr to rent. Many models are sufficiently complicated (not just large) that you end up training them for days and weeks, even with this much compute. So the numbers add up quickly.

But just being “big” doesn’t mean “trained for a month on a supercomputer”.

[1] https://cloud.google.com/tpu

[2] https://www.google.com/amp/s/cloudblog.withgoogle.com/produc...

Re: AI Democratization in the Era of GPT-3

#48

Post by 2sk21 »

exged wrote: post #41
anoncareer0212 wrote: post #33

Earlier quoted context omitted.

Given the worst-case assumptions of people arguing the cloud is required[1], it's a one-time investment of $60,000, $200,000 in the absolute worst case. [1] https://news.ycombinator.com/item?id=24601264

That's for model inference. For training OpenAI said they used around 3000 PetaFLOPS / days on the largest GPT-3 model. That translates to about 300 Nvidia A100 GPUs if you want training to finish in a month (any slower and your researchers are not going to be able to make much progress). A system like that would cost at least $5M, probably more like $10M.

Surely $10 million should be well within the spending abilities of a fair number of tech people? Many universities now have fairly large computing clusters as well.

Re: AI Democratization in the Era of GPT-3

#49

Post by r7r73hdhdu »

jackcosgrove wrote: post #39

The PC was supposed to democratize computing. It did, and then Microsoft found a choke point. The internet was supposed to democratize communication. It did, and then Google found a choke point. It seems like this two steps forward, one step back pattern might be the rule rather than the exception. Even the article defines AI democratization in terms of using models rather than training models, as the costs of traini…

This seems like the most cynical take possible, most of those choke points aren't really that firm anymore. Microsoft had a brief chokehold and then smartphones became the computer of the working class. Google is dominant in search but now they're competing against other megacorps who are investing heavily in catching up. All that's died is the naivety of the 80s and 90s that believed small mom and pops would somehow be outcompeting megacorps rather than serving as product development for them.

Re: AI Democratization in the Era of GPT-3

#50

Post by Fordec »

I applied for API access months ago and still haven't been able to tinker with it.

Until it's immediate sign up and not some insider walled garden waitlist, from a developer perspective, OpenAI and GPT-3 is anything but Open.

If I was overly pushed to get access, my best chance of success right now appears to be a search on Github for somebody elses leaked API key in a repo somewhere.

Post reply on HN