Natural language benchmarks don’t measure AI models’ general knowledge well

Judea Pearl has been bringing up the lack of causal knowledge in ML very often. He has even posted lots of interesting comments in Andrew Gelman's blog, e.g.: https://statmodeling.stat.columbia.edu/2009/07/05/disputes_a...

I tend to think that lots of solutions could come from topics like those discussed in this book, with a lot of further development: http://probmods.org/

It seems like there are not many common-knowledge questions that you can't find answers to on the Internet? Most of us rely on web search these days.

Also, making up answers isn't necessarily a good skill to train for if accuracy is needed. It would be more useful to quote and cite the source.

It does lead to intelligence and knowledge of the world. It leads to somewhat similar understanding of the world as our understanding of 4D, quantum mechanics or environment close to a black hole. Pretty bad one, mostly based on repeating whatever we have read about in popular science literature without any intuitive understanding. GPT-3 understanding of physical world is similarly bad. It's just simply a modality not available to the model.

But GPT-3 is able to do few-shots learning. If you show it couple of times how to do something it will try to repeat after you showing some rudimentary understanding of what you are trying to do. It's even able to learn to do simple analogies [0].

GPT-3 is able to do very, very simple arthritics like addition and subtractions with 2-3 numbers. Actually even a lot smaller models trained on arthritics are able to do impressive feats like symbolic integration and solving differential equations.

> Deep Learning for Symbolic Mathematics [1]: Neural networks have a reputation for being better at solving statistical or approximate problems than at performing calculations or working with symbolic data. In this paper, we show that they can be surprisingly good at more elaborated tasks in mathematics, such as symbolic integration and solving differential equations. We propose a syntax for representing mathematical problems, and methods for generating large datasets that can be used to train sequence-to-sequence models. We achieve results that outperform commercial Computer Algebra Systems such as Matlab or Mathematica.

What is true is that GPT-3 is not very good at following instructions. It would not be able to execute an algorithm. So, it would not learn arithmetic from instructions if you would not include explicit examples.

[0] https://medium.com/@melaniemitchell.me/can-gpt-3-make-analog... [1] https://arxiv.org/abs/1912.01412

It is very true that GPT-XYZ is fundamentally limited, like all transformer models, but don't misattribute the root cause to the fact that it's just processing text.

The real limitation is that these are feedforward networks that just do perception without any processing of what they've perceived. You can try to hide that fact for a while by increasing the depth of the perception network, basically hard-coding some processing into the single pass, but you're still not capturing any of the "absolutely requires self-feedback" behavior that we care about that takes a human more than a split second to do (aka almost all actual thought).

A statistical model that made good use of what was coming in could absolutely learn (to take your example) math at every level based on nothing but text, transformers are just nowhere close to having that capability because of their design limitations.

The article doesn't give examples, but the paper does. Here are a few examples of question overlap, for context.

    Test Question:  who plays max voice in a goofy movie
    Train Question: who does max voice in a goofy movie
    Answer:         Jason Marsden

    Test Question:  when will the 2018 oscar nominations be announced
    Train Question: when are the oscar nominations for 2018 announced
    Answer:         January 23 2018

    Test Question:  who has scored more goals in the premier league
    Train Question: most goals scored by a premier league player
    Answer:         Alan Shearer

    Test Question:  where are the cones in the eye located
    Train Question: where are cone cells located in the eye
    Answer:         retina

    Test Question:  who led the conquest of the incas in south america
    Train Question: conquistador who defeated the incan empire in peru
    Answer:         francisco pizarro

It makes sense that questions like these are especially susceptible to brute memorization. These are certainly bugs in the benchmark—and, in fact, about a third of questions in each dataset have this issue.

The paper also considers answer overlap; that is, when the answer to the question also occurs in the training set. This alone does not imply memorization, but it does open the door for some shortcuts. Some examples from the paper are:

    Open Natural Questions
    Duplicated: Phil Simms, Brian Johnson, 8, the Indians, the 1830s
    Unique:     Cloves, Matt Monro, 1,020 – 1,080 kg, Hermann Ebbinghaus, Matt Flinders

    TriviaQA
    Duplicated: David Bowie, Battle of camlann, Heligoland, Henry VII, Niagra Falls
    Unique:     Death in the afternoon, Clash of the Titans, ice-cream sundae, Camshaft, Cumberland

    WebQuestions
    Duplicated: Harvard, Alderaan, India, 2011, Zeus
    Unique:     Queen Victoria, Braslia, Paddington, Tom Corbett, Gary

It's less obvious how to treat answer overlap. In particular, removing answer overlap might bias the dataset towards harder questions. Some of the reduction in model scores is presumably because models have to work harder to understand the question, as merely looking for topical similarities will be ineffective, but it also removes questions whose answers are general enough to apply to many questions, like ‘8’, ‘the 1930s’, ‘Harvard’, ‘2011’, etc. This means that reductions in the score don't clearly say how much cheating happened. However, the success of the BERT-based Nearest Neighbor model, which retrieves the answer of the most semantically similar fine-tuning sample, scores similarly to BART, which seems much too high for comfort.

It should be noted that a lot of the discussions about causal knowledge and intelligence aren't too relevant to these questions, as they are largely tests of memory or retrieval. It is expected that these models answer the questions by searching their training data or their document index. The issue is that their training data isn't meant to contain copies of the test questions, just the information sufficient to answer them.