GPT-3 Is “Mindblowing” If You Don’t Question It Too Closely

Tired lame criticisms.

For example, Bender's award-winning paper might impress you less if you knew that GPT-3 already solved their counterexamples which supposedly demonstrated what language models will never be able to do: https://www.gwern.net/GPT-3#bender-koller-2020

Van den Broeck is just wrong, there's a lot of incredible scientific value in the GPT-3 work. The meta-learning, which is the main result of the paper, is itself a landmark finding, plus all the bonus material about scaling curves. Dismissing that is a little like dismissing finding the Higgs because 'we all knew the Higgs existed, it just shows how much money the EU was willing to throw at the LHC'. Good grief.

And Kevin Lacker's post, which I am apparently doomed to see cited endlessly, is less than meets the eye; many of Lacker (and Shane's) failure cases can be solved with better prompts and sampling settings: https://www.gwern.net/GPT-3#common-sense-knowledge-animal-ey... and following sections. (I hadn't tested the prompts about 'what number comes before one thousand' etc, but testing the 10,000 one right now, better sampling fixes that one as well.)

I remember an article which showed that GPT-3 quality of answers were very dependent of the previous questions which I find quite surprising as it has already been trained on a large number of texts..

And there were an attempt to train GPT-2 with a common sense database: https://www.quantamagazine.org/common-sense-comes-to-compute... it would be interesting see the results with GPT-3..