AI Democratization in the Era of GPT-3
thegradient.pub
AI Democratization in the Era of GPT-3
1–10 of 90 posts
Re: AI Democratization in the Era of GPT-3
#2I do agree that it isn’t “democratization” in the sense of “having access” and that it’s a buzz word. But with Azure Cognitive Services and Azure AutoML, I do think MSFT has shown itself capable of point 3: enabling its customers “to use the algorithms and models, potentially without requiring advanced mathematical and computing science skills”.
Re: AI Democratization in the Era of GPT-3
#3Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?"
It seems like for super gigantic models, good infra to train on cloud deployment (for which many labs can have credits / grants) would be the first priority, followed perhaps by good compression/pruning tech so after training inference can be done on one/fewer GPUs. Any other things?
Re: AI Democratization in the Era of GPT-3
#4OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or information available such as what he asks about.
But I personally find the complaints to be understandable, especially as someone that didn't get a response for my requests for GPT-3 beta access, it felt pretty bad to watch everyone else have fun building cool things with the world's best text AI while I sat there and couldn't do anything, even if I was willing to pay for access.
Hopefully there will be other relevant players here besides just OpenAI sooner or later.
Re: AI Democratization in the Era of GPT-3
#5OpenAI's goal is definitely not to give everyone unlimited/equal access to powerful tools like GPT-3. We've had countless jokes about the name being 'OpenAI', and perhaps it's true that it's not the best name (along with 'democratizing' AI), but I'm not sure the author is suggesting a solution here rather than just venting that things seem kind of unfair, and no one outside of OpenAI really has much control or inform…
"other relevant players" -> Google could create one.
Re: AI Democratization in the Era of GPT-3
#6Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deploy…
Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?
Re: AI Democratization in the Era of GPT-3
#7Curious as to what people think wrt the final question " Do we trust [OpenAI] to take on this role [of deciding who can have access to newer SOTA models and who can't]? And if not, how can academics and practitioners fight for the continued democratization of AI, as some of its most important techniques become as hard to replicate as GPT-3?" It seems like for super gigantic models, good infra to train on cloud deploy…
Maybe the authors of the text should also have a stake in GPT-3. After all, OpenAI didn't write the corpus. Google benefits from the web, and Facebook from the real-life social networks and their activities that it replicates online (messages, meetings, news, etc). We are all the source of the training data. Why should we be at the whims of these derivative product companies?
Fun fact, OpenAI did not collect the data itself - they mainly used data from Common Crawl (in addition to a couple other datasets), which is compiled by a non profit that shares the dataset for free. So perhaps the license of such datasets can encourage free sharing of research outcomes.
Re: AI Democratization in the Era of GPT-3
#8Forget AI democritization, GPT-3 is AI demoguerization.
GPT-3 is singular; it is one model, one dataset, one training. Yet it will be the only one that will exist for quite some time (or by far the most available), and now it will underwrite productization and malfeasance, a la mode pay to play.
For example, I recently read a paper supposedly written by a Chinese dissident virologist, which report was disseminated by a group with questionable membership. Most of the jargon in the report going over my head, I had to wonder if the otherwise convincing verbiage wasn't the work of GPT-3.
Re: AI Democratization in the Era of GPT-3
#9Disclosure: I work on Google Cloud and have worked with the OpenAI folks on large models.
This article mixes both “should research be open” and “is this work cheaply reproduced / accessible”:
For smaller, open models:
> The average person could not recreate models of this size from scratch, but the models can run on a single machine with a single GPU.
but about GPT-3:
> GPT-3 represents a new circumstance. For the first time, a model is so big it cannot be easily moved to another cloud and certainly does not run on a single computer with a single or small number of GPUs. Instead OpenAI is providing an API so that the model can be run on their cloud.
While I’d quibble with “for the first time” (it’s easy to generate mega models! Plenty of distributed mesh tensorflow stuff does that, etc.), I don’t think this is any different than large physics simulations.
Is it “wrong” to have some groups push the boundary of what’s possible with supercomputers? I certainly don’t think so. If anything, it shows what’s possible and others can do the valuable work of “miniaturization”. In this specific area, PRADO is a good example relative to BERT. For my historic area of ray tracing research, we did lots of things on an SGI Origin that let us “jump ahead a few years” versus what we could have done on any basic workstation.
You could argue that it’s not academically interesting (“you just ran this really big because you had the budget / hardware, whatever”) and reject the paper. [Edit: I consider this kind of work interesting from a systems perspective, not “ML”, but it’s still interesting!] I don’t think it makes sense to suggest that we should hold back progress based on NSF grant funding or least common denominator computing resources. How would you decide what’s acceptable? Is a single A100 affordable? Only a T4? Only a laptop?
tl;dr: it’s fine to argue about openness and democratization being hollow marketing words. I’m not sure I would conflate openness with “everyone, everywhere should be able to run any scientific work without expense”.
Re: AI Democratization in the Era of GPT-3
#10Powerful models are trained using large amounts of data that most people do not have access to.
They are also trained powerful infrastructure that most people do not have access to.
So, to speak of democratization is interpreting the current state of affairs incorrectly.