I think the main disappointment is that we humans aren't that special when a brute-forced scalable transformer is getting into our ballpark. We have also recently seen how Open AI + MS were able to use a GPT-variation for automated text-description-to-python-code generation, and utilizing something like GPT-3 in that task might render many swengs obsolete fairly soon.
GPT-3: A Disappointing Paper?
51–60 of 87 posts
Re: GPT-3: A Disappointing Paper?
#52It would be cool if there was a platform to crowd source compute resources to train stuff like this so that regular people (without 7 figure budgets) can have access to these models which are becoming increasingly out of reach to the general public.
Here is a recent paper (disclaimer: I am the first author) named "Learning@home" which proposes something along these lines. Basically, we develop a system that allows you to train a network with thousands of "experts" distributed across hundreds or more of consumer-grade PCs. You don't have to fit 700GB of parameters on a single machine and there is significantly less network delay as for synchronous model parallel training. The only thing you sacrifice is the guarantee that all the batches will be processed by all required experts.
You can read it on ArXiv https://arxiv.org/abs/2002.04013v1 or browse the code here: https://github.com/learning-at-home/hivemind. It's not ready for widespread use yet, but the core functionality is stable and you can see what features we are working on now.
Re: GPT-3: A Disappointing Paper?
#53All valid points, but I disagree with the conclusion, for several reasons: * First of all, the GPT-3 authors successfully trained a model with 175 billion parameters. I mean, 175 billion. The previous largest model in the literature, Google’s T5, had "only" 11 billion. Models with trillions of weights are suddenly looking... achievable. That's a significant experimental accomplishment. * Second, the model achieves co…
I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his r…
You have misunderstood Sutton's argument.
Quoting Rich:
One thing that should be learned from the bitter lesson is
the great power of general purpose methods, of methods that
continue to scale with increased computation even as the
available computation becomes very great.
Re: GPT-3: A Disappointing Paper?
#54Earlier quoted context omitted.
I don't know why this reminds you of Sutton's essay. Sutton claims that algorithmic work will be superseded by brute-force approaches powered by exponential growth in computing power. But this paper is not the result of exponential growth in computing power. It's the result of $1b worth of Azure credits provided for free by Microsoft. Rich Sutton is a great scientist but he is fooled by randomness. He initiated his r…
The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.
Yes then credit to azure engineers not open ai people
Re: GPT-3: A Disappointing Paper?
#55I could not disagree more with this post. To summarize what the author is unhappy with: 1) "It’s another big jump in the number, but the underlying architecture hasn’t changed much... it’s pretty annoying and misleading to call it “GPT-3.” GPT-2 was (arguably) a fundamental advance, because it demonstrated the power of way bigger transformers when people didn’t know about that power. Now everyone knows, so it’s the f…
I think we should not compare if anybody is capable or not. Here the most of the work is done by azure engineers and none of them are mention in the paper. So no even open ai can’t do it without azure infrastructure.
Re: GPT-3: A Disappointing Paper?
#56My biggest problem with GPT3 is that it's not going to be accessible (practically speaking) to the general public. There's been a recent push to democratize this type of work with libraries like Huggingface transformers, but models this large will force the benefits of this work back into the ivory tower.
Re: GPT-3: A Disappointing Paper?
#57Earlier quoted context omitted.
The engineering effort of utilizing 1B worth of Azure credits is surely worth at least 10% of it.
Yes then credit to azure engineers not open ai people
Wow, do you reasonably believe OpenAI people are not contributing?...
I thought HN visitors are not capable of that level of naive thinking.
Re: GPT-3: A Disappointing Paper?
#58Re: GPT-3: A Disappointing Paper?
#59Earlier quoted context omitted.
I mostly agree with you. However, > It's a significant experimental accomplishment that points to a future in which general-purpose NLP models could be used for novel tasks without requiring additional training from the get-go. This premise is still purely science fiction. This model does not touch on either novel tasks nor being free from pretraining (unless I misunderstand). But overall, I think you’re right: it’s…
GPT-3 was pretrained on five datasets (Common Crawl, WebText2, Books1, Books2, and Wikipedia; see table 2.2), and then used on previously unseen tasks (Q&A, translation, cloze, etc.) without finetuning , i.e., weights were not updated after the original (autoregressive) pretraining. This promises a possible future in which general-purpose models are pretrained once, and deployed to production for multiple tasks.
This is pretty mind-boggling, to the point of suspecting an error in the methodology. If it is just a language model, then it has no baked in notion of test time tasks. How on earth does a language model know what is required of it at test-time without fine tuning? How does it know that the test time prompt are examples of the task, and not some story prompt it's supposed to riff off in random ways?
Re: GPT-3: A Disappointing Paper?
#60Earlier quoted context omitted.
I've seen you post this before and you seem enthusiastic about it. Do you have a good summary of why you feel this paper so important?
why you feel this paper so important? the best summary are the benchmarks: first place at question answering (on yahoo task) First place on language modeling for pen treebank (by FAR) So it is a totally new model that will probably keep evolving and being applied to more and more kind of NLP tasks. And it seems that it can have the first place on most NLP tasks, its empirically the breakthrough of the year. It achiev…
Not all models scale up well with more parameters.
VAEs are not really exclusively a vision thing, they have been used in a variety of settings. Using VAEs for NLP is also nothing new, an early example is Bowman et al, 2015.