Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Democratization in the Era of GPT-3

thegradient.pub

Post reply on HN
Print view View original

21–30 of 90 posts

Re: AI Democratization in the Era of GPT-3

#21

Post by mirekrusin »

29athrowaway wrote: post #10

Powerful models are trained using large amounts of data that most people do not have access to. They are also trained powerful infrastructure that most people do not have access to. So, to speak of democratization is interpreting the current state of affairs incorrectly.

They trained it on freely available datasets managed by nonprofits as somebody mentioned.

Powerfull infrastructure can be democratized in similar way that SETI@home, cryptocurriencies and other P2P projects do it.

In theory this can be done.

Re: AI Democratization in the Era of GPT-3

#22

Post by hanoz »

gdb wrote: post #14
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

(I work at OpenAI.) > especially as someone that didn't get a response for my requests for GPT-3 beta access We are still working our way through the beta list — we've received tens of thousands of applications and we're trying to grow responsibly. We will definitely get to you (and everyone else who applies), but it may take some time. We are generally prioritizing people with a specific application they'd like to b…

> We are generally prioritizing people with a specific application they'd like to build.

Why?

Re: AI Democratization in the Era of GPT-3

#23

Post by ve55 »

gdb wrote: post #14
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

(I work at OpenAI.) > especially as someone that didn't get a response for my requests for GPT-3 beta access We are still working our way through the beta list — we've received tens of thousands of applications and we're trying to grow responsibly. We will definitely get to you (and everyone else who applies), but it may take some time. We are generally prioritizing people with a specific application they'd like to b…

Thanks for the response - I had assumed the beta period was soon coming to an end, so by the time I was able to have access I'd have to pay just for basic experimentation. It was hard to say specifically what I'd design since I'd have to experiment with the API first to see if the ideas I had were feasible, so I probably did a poor job at that part of the application, but appreciate the offer!

Re: AI Democratization in the Era of GPT-3

#24

Post by mensetmanusman »

I learned some things.

GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run.

Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals.

If this is true, OpenAI will not be able to move to the next level without partnering with very wealthy institutions a la Microsoft.

If this is true, those calling for OpenAI to not monetize intermittent progress are essentially preventing next generation discovery, unless they have alternative monetization ideas to generate 8 figures for research.

Re: AI Democratization in the Era of GPT-3

#25

Post by csande17 »

ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

Surely the solution here is "put the model on BitTorrent, you cowards".

Like, okay, the model's big and unwieldy to run. But hardware's always getting better, and there are lots of research use-cases where it's okay if it takes ten minutes to page the model in and out of SSD while generating predictions. Plus, maybe we'd get some more discoveries in the field of efficiently running huge models.

The arguments about "safety" were PR nonsense when they were making them about GPT-2, and they're nonsense now. It's a robot that blends up Reddit posts in a food processor, it's barely more advanced than tapping the iPhone predict-next-word button over and over, it's not going to hack the Pentagon or take over the world. The only reason OpenAI has ever had to not publish their models -- and I am ashamed that this industry doesn't call them out more often on this -- is so that they can generate positive press coverage on launch day with unrefutable cherry-picked examples.

Re: AI Democratization in the Era of GPT-3

#26

Post by [deleted] »

ma2rten wrote: post #12
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

The article is hinting at this but I also think many people who complain that OpenAI didn't release the model don't understand how big this model actually is. Even if they had access to the parameters they couldn't do much with it. Assuming you used single precision the model is 350 gigabytes (175 billion * 2 bytes). For fast inference the model needs to be in GPU memory. Most GPUs have 16GB of memory, so you would n…

[deleted]

Re: AI Democratization in the Era of GPT-3

#27

Post by boulos »

csande17 wrote: post #25
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

Surely the solution here is "put the model on BitTorrent, you cowards". Like, okay, the model's big and unwieldy to run. But hardware's always getting better, and there are lots of research use-cases where it's okay if it takes ten minutes to page the model in and out of SSD while generating predictions. Plus, maybe we'd get some more discoveries in the field of efficiently running huge models. The arguments about "s…

The author though raises concerns about both the availability (openness) of the model as well as the current ability to run it due to cost (equity / equality of access). Making the model available would still not make it equal access.

I’m not saying I agree or disagree with the openness argument, but the equality argument is separate.

Re: AI Democratization in the Era of GPT-3

#28

Post by claudeganon »

csande17 wrote: post #25
ve55 wrote: post #4

OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…

Surely the solution here is "put the model on BitTorrent, you cowards". Like, okay, the model's big and unwieldy to run. But hardware's always getting better, and there are lots of research use-cases where it's okay if it takes ten minutes to page the model in and out of SSD while generating predictions. Plus, maybe we'd get some more discoveries in the field of efficiently running huge models. The arguments about "s…

It’s beyond absurd to build a model with public, user-created data, gathered and released for free by a nonprofit, and then claim “uh, the model’s too big and unwieldy, so we have to keep things under lock and key.”

I don’t doubt that they’ll profit handsomely from this approach, but it’s the height of cynicism to engage in this kind of stuff and their statements around the practice should be taken in kind.

Re: AI Democratization in the Era of GPT-3

#29

Post by choppaface »

mensetmanusman wrote: post #24

I learned some things. GTP-3 is so complex the model requires large cloud computing resources to run. Ergo, it is also very expensive to run. Assumption: Bleeding edge AI will require tens of millions of dollars of computation before new network architectures fall out of state space. After this, the models can be pruned to be ran by mere mortals. If this is true, OpenAI will not be able to move to the next level with…

There will always be models and research that require proprietary hardware and resources and thus cannot be “democratized.” PageRank required a ton of hardware and an index. Non-industry researchers will always be priced out of something.

A key concern here with regards to the ethics-of-AI issue is that last year OpenAI refused to release GPT-2 because it was too ‘dangerous’. This year, GPT-3 is suddenly a revenue-generating Microsoft product. The Gradient article linked is one of the more diplomatic ways of calling BS on OpenAI’s strategy. Financial interests taint research in subtle ways, it has for decades, and OpenAI employees being paid $1m in cash salary need reminders of this fact.

https://openai.com/blog/better-language-models/

Post reply on HN