Natural language benchmarks don’t measure AI models’ general knowledge well

Hi, first author here.

> Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?)

I think this is definitely the case for the BART model. It is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time. Note that the T5-11B+SSM closed-book model was able to do a little better on NQ, so very large models with task-specific pretraining objectives do seem to do something slightly more interesting than just NN, but still really struggle in some settings.

> And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this?

Great question! Converting sentences into a series of QA pairs is something we're really interested in. The T5-11B+SSM model we evaluate in the paper uses a special "Salient span masking" pretraining objective that does this to some extent (only mask words at pretraining time that are likely to be "answers" to factual questions), so in essence the pretraining task becomes pretty standard cloze-question answering, and they find that leads to better downstream results (https://arxiv.org/abs/2002.08910)

I agree that causal inference certainly requires ability to interact with the environment. In do-calculus this follows basically from the definition of the "do" operator.

As other people already mentioned, the "do" operator does not need to be related to the physical human scale environment. Causal inference could be useful also in interacting with the internet or other virtual environments like games.

Additionally, even humans do not have intuitive understanding of physical reality outside of our evolutionary environment. Nobody is able to intuitively understand quantum mechanics or general theory of relativity. Our intuitions of causality can beak even in relatively mundane environment like low Earth orbit. For example, you can fire thrusters to push you towards an object and still miss that object. There are actual missions in LEO that failed do to this type of mistakes of elite test pilots employed by NASA.

Of course, you can use mathematical formalism to reason about unfamiliar environments, but no matter how much time you spend learning about multidimensional spaces you will not be able to imagine 4D space or a quantum wave. But nobody is claiming that we do not understand language like "spin of an electron" because we have never experienced what a spin of an electron is.

Now to get to my original point. My original point was that causal inference is not a basic building block of intelligence. Formalism like do-calculus or logic are too brittle to be a building block of intelligence. You need something that is robust to noise. Something that can consume a tensor of pixels and process it. Once you have a system that is able to deal with this type of inputs then you can strive to do something like do-calculus or logic on top of it. But my argument goes further. My guess is that ability to do formal reasoning will emerge from the need to carry out complex tasks. Nobody will intentionally program models to do logic or do-calculus. Ability to do it will arise in models as a combination of the necessity of solving complex tasks and clever training techniques as it did trough evolutionary process in humans.

PS.

> So no matter how much you are able to reproduce these patterns, they contain no content. The content is in the reader.

To me this type of assertions do not contain any content.

"The content is in the reader." I don't understand that at all. Maybe I'm just a Chinese room :) .

Approximately, the semantics of natural language are concepts. Concepts in this sense are deep bodily-inferential-causal models.

If I say, "No! The glass is under the desk" you can immediately: find the glass, reason about why you havent previously found it, ask for the glass to be filled, ask for the glass to be handed to you, report on whether you like where the glass is..

You could say, "Why is the glass on the floor!?" angrily.

And by doing so communicate an expectation about where glasses ordinarily are, express a desire-frustration, express a confusion over the intentions of others..

The information contained in "No! The glass is under the desk" is VAST.

It is vast because our understanding of the world is vast. Not because the sentence as a lot of letters. Not because the words in it occur in a certain frequency. Nothing about the sentence itself is vast. It is profoundly shallow.

Intelligence is this vastness. Intelligence is moving effortlessly from "why" to "how" to "when" to "what" across domains, across hypothetical/counterfactual scenarious, across intentions/expectations.

This vastness is laden in animal minds, even, essentially constitutive of animal minds. And it cannot be found in "data".

It is grown by the complex biophysical process of learning, which requires a deep (& playful) interaction with the environment. The environment grows your capacities as you play with it.

>Causal analysis can be performed, via Pearl, on datasets collected for causal analysis.

There's a lot of past and current work on causal inference on observational data too.

If a text corpus correlates with human knowledge (your "presupposes a certain model of the world"), then it necessarily encodes the portions of the knowledge it correlates with. That is, correlations drive the search of model parameters that explain the correlations. The stronger or more numerous the correlations, the more accurate a model can be computed. But the search space informed by this network of correlations is much more restricted than a naive brute force search, thus the correlations provide information about, i.e. encodes, the correct model.