Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Democratization in the Era of GPT-3

thegradient.pub

Post reply on HN
Print view View original

11–20 of 90 posts

Re: AI Democratization in the Era of GPT-3

#11

Post by searchableguy »

andreyk wrote: post #7
visarga wrote: post #6

Earlier quoted context omitted.

Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?

Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes. https://commoncrawl.org/

Shouldn't I own part of gpt3 because it's trained on my data and likely spitting out what I have commented somewhere on the internet?

I think ml models should be public unless the data itself isn't.

Re: AI Democratization in the Era of GPT-3

#12

Post by ma2rten »

ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it.

Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would need 22 GPUs just to hold the model in memory and that doesn't even include the memory for activations.

If you wanted to do fine tuning, you would need 3x as much memory for gradients and momentum.

Re: AI Democratization in the Era of GPT-3

#13

Post by tarboreus »

ma2rten wrote: post #12
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would n…

If it was open, there would be other services offering this, and not just an opaque beta and now a single expensive service.

Re: AI Democratization in the Era of GPT-3

#14

Post by gdb »

ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

(I work at OpenAI.)

> especially as someone that didn't get a response for my requests for GPT-3 beta access

We are still working our way through the beta list — we've received tens of thousands of applications and we're trying to grow responsibly. We will definitely get to you (and everyone else who applies), but it may take some time.

We are generally prioritizing people with a specific application they'd like to build, if you email me directly (gdb@openai.com) I may be able to accelerate an invite to you.

Re: AI Democratization in the Era of GPT-3

#15

Post by wslh »

ma2rten wrote: post #12
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would n…

Tell cryptocurrency miners that this is a big model to compute... the size of this problem seems very tiny.

If there are millions of ASIC, GPU, etc devices mining cryptocurrencies it is fair to speculate that democratizing AI has a special room in this model.

Re: AI Democratization in the Era of GPT-3

#18

Post by whichquestion »

andreyk wrote: post #3

Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deploy…

Would we be able to democratize these large ML models using something like SETI@Home or similar? Would more research in distributing models across large networks be useful? Has research on this topic been done?

Re: AI Democratization in the Era of GPT-3

#19

Post by liuliu »

ma2rten wrote: post #12
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would n…

You need tiny bit of memory for activations if you don't want fine-tuning. I think for GPT-3, fine-tuning is out of window. But it is reasonable to expect inference takes less than a minute with single 3090 and fast enough SSD.

Re: AI Democratization in the Era of GPT-3

#20

Post by boulos »

29athrowaway wrote: post #10

Powerful models are trained using large amounts of data that most people do not have access to. They are also trained powerful infrastructure that most people do not have access to. So, to speak of democratization is interpreting the current state of affairs incorrectly.

See my sibling comment, but the dataset that the GPT-2 folks did was “just grab whatever people on reddit upvoted”. Any grad student might have done the same (and in a sense, James Hays and Alyosha Efros did something like that long ago with flickr data for im2gps).

While GPT-3 is trained on a massive V100 cluster, you could probably do so with a much smaller one / there exist interesting smaller models. It’s expensive to rent this class of equipment, but it is available.

The distinction is that OpenAI made a focused bet. Most research funding and labs spread their bets heavily (e.g., each institution or researcher gets $50k/yr of funding). OpenAI takes a different stance, but it’s not clear that they’re spending even as much as say Google, Facebook or other large institutions. It’s also not obvious that you even have to play the same game to get similar results.

Post reply on HN