Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Natural language benchmarks don’t measure AI models’ general knowledge well

venturebeat.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#61

Post by mjburgess »

hackinthebochs wrote: post #60
mjburgess wrote: post #52

Earlier quoted context omitted.

So this is to repeat hume and mistake causal analysis as a kind of induction or inference. It isnt. Our bodies are the primary site of "causal analysis". Eg., when I touch the hot surface of a stove I do not infer that temperature is a cause of my hand being hot. Such an inference, as the basis for our models of the world, would be -- as you/hume/etc. say -- deeply insufficient. The operation of the world on our bodi…

None of this explains how modelling time sequences of events does not sufficiently approximate the kind of causal knowledge you mention.

Suppose I place a pot on a stove and the water boils.

Now I feed into the machine what data?

Here, it gets this: all gravitational, electomagnetic, etc data within 1km; all geometric information about all objects within 1km (, and all of their properties, etc.)

Now, machine, what caused the pot to boil?

It has no clue. There are an infinite number of antecedent temporal events.

The problem isn't the mathematics of causal inference. The problem is relevance. That isn't an inferential problem.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#62

Post by tsimionescu »

hackinthebochs wrote: post #59
tsimionescu wrote: post #34

Earlier quoted context omitted.

As another poster wrote, the main problem is not the neural network itself, it is the training set, and very likely the training algorithm as well. My claim is that working knowledge of the world is simply not encoded in human text. All human text (and speech) presupposes a certain model of the world that is learned from experiencing the world, likely based on a model that itself is partially pre-trained by evolution…

If a text corpus correlates with human knowledge (your "presupposes a certain model of the world"), then it necessarily encodes the portions of the knowledge it correlates with. That is, correlations drive the search of model parameters that explain the correlations. The stronger or more numerous the correlations, the more accurate a model can be computed. But the search space informed by this network of correlations…

Then I used the wrong words. My point is that human text is essentially encrypted using a particular model of the world as the encryption/decryption key. You can't derive the key from the cipher text.

For example, if I give you the text 'the car is red', there is no watly to imagine the meaning of this phrase (in any sense of the word 'meaning') unless you have some understanding of what cars are, what red is, and what it means for a car to be red (which is somewhat different from what it means for an apple to be red, for example).

It is interesting that with enough statistical analysis you can begin to decipher certain parts of the true message without knowing the cipher key. But this only works for those parts where you have a large corpus of text talking about them, and there are many common human needs or experiences that do not have such a corpus. It is also very easy to capture answer/response pairs in the corpus and repeat them, seeming to have high accuracy, but in fact failing to generalize more than meets the eye.

For example, with enough compute power, you could build a machine that, for any query, would find the most similar such query in all of its training data, and respond with the continuation of that query (which could be an answer or a follow-up question). This could be quite impressive in conversation, while doing nothing to elucidate anything about human intelligence, and being wholly unable to adapt to any novel situation.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#63

Post by hackinthebochs »

mjburgess wrote: post #61
hackinthebochs wrote: post #60

Earlier quoted context omitted.

None of this explains how modelling time sequences of events does not sufficiently approximate the kind of causal knowledge you mention.

Suppose I place a pot on a stove and the water boils. Now I feed into the machine what data? Here, it gets this: all gravitational, electomagnetic, etc data within 1km; all geometric information about all objects within 1km (, and all of their properties, etc.) Now, machine, what caused the pot to boil? It has no clue. There are an infinite number of antecedent temporal events. The problem isn't the mathematics of ca…

>It has no clue. There are an infinite number of antecedent temporal events.

Right, determining the cause of the pot boiling given only local information about this one event is impossible. But that's not a good representation of learning from a real-world data set. A real training corpus might also have an example of a pot boiling in a campfire, a water heater heating up water using fire, someone touching an open flame and going "ouch!", fire burning down a house, etc. All these examples taken together can reasonably lead one to infer that fire causes things to become hot. This is the kinds of regularity found in real world datasets and a good learning algorithm will extract this regularity in the course of predicting the dataset.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#64

Post by hackinthebochs »

tsimionescu wrote: post #62
hackinthebochs wrote: post #59

Earlier quoted context omitted.

If a text corpus correlates with human knowledge (your "presupposes a certain model of the world"), then it necessarily encodes the portions of the knowledge it correlates with. That is, correlations drive the search of model parameters that explain the correlations. The stronger or more numerous the correlations, the more accurate a model can be computed. But the search space informed by this network of correlations…

Then I used the wrong words. My point is that human text is essentially encrypted using a particular model of the world as the encryption/decryption key. You can't derive the key from the cipher text. For example, if I give you the text 'the car is red', there is no watly to imagine the meaning of this phrase (in any sense of the word 'meaning') unless you have some understanding of what cars are, what red is, and wh…

Your encryption analogy doesn't really work either. Encrypted text encodes the key used to encrypt it. This is why it takes 2^N computational steps to recover the plain text, where N is the key size rather than where N is the text size. It's just that the computational cost of decrypting with sufficiently large keys is intractable.

But human languages have much more exposed regularity than ciphertext from good encryption algorithms, so the computational cost is much lower. The "key", i.e. world model, in case of human languages is plausibly much bigger, but since human languages have significant exposed regularity, it is still tractable to compute, with the trade off that you need much more text to do it.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#65

Post by mjburgess »

hackinthebochs wrote: post #63
mjburgess wrote: post #61

Earlier quoted context omitted.

Suppose I place a pot on a stove and the water boils. Now I feed into the machine what data? Here, it gets this: all gravitational, electomagnetic, etc data within 1km; all geometric information about all objects within 1km (, and all of their properties, etc.) Now, machine, what caused the pot to boil? It has no clue. There are an infinite number of antecedent temporal events. The problem isn't the mathematics of ca…

>It has no clue. There are an infinite number of antecedent temporal events. Right, determining the cause of the pot boiling given only local information about this one event is impossible. But that's not a good representation of learning from a real-world data set. A real training corpus might also have an example of a pot boiling in a campfire, a water heater heating up water using fire, someone touching an open fl…

Who prepares the dataset?

You are just shifting relevance problem to the human to solve. That's my point.

The "regularity" isn't very hard to find when the data which exposes it is already selected.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#66

Post by hackinthebochs »

mjburgess wrote: post #65
hackinthebochs wrote: post #63

Earlier quoted context omitted.

>It has no clue. There are an infinite number of antecedent temporal events. Right, determining the cause of the pot boiling given only local information about this one event is impossible. But that's not a good representation of learning from a real-world data set. A real training corpus might also have an example of a pot boiling in a campfire, a water heater heating up water using fire, someone touching an open fl…

Who prepares the dataset? You are just shifting relevance problem to the human to solve. That's my point. The "regularity" isn't very hard to find when the data which exposes it is already selected.

No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorithm or the training data you are suggesting. You haven't given any reason to think otherwise.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#67

Post by nl »

patrick-lewis wrote: post #51
nl wrote: post #21

Earlier quoted context omitted.

Nice paper and important work. Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) And if this is the case do you think training-time processing of data to attempt to move convert it to question/answer form data rather than raw QA would be a reasonable approach towards tackling this?

Hi, first author here. > Given that nearest-neighbor outperforms on closed book, is it reasonable to suspect the model is doing NN itself internally (which would explain the good performance on close duplicates?) I think this is definitely the case for the BART model. It is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time…

Thanks for the response.

> [BART] is essentially acting as a QA-pair memorizer over the training data, and at test time, it just matches the question onto those seen at training time.

Not super-surprising.

> The T5-11B+SSM model we evaluate in the paper uses a special "Salient span masking" pretraining objective that does this to some extent (only mask words at pretraining time that are likely to be "answers" to factual questions), so in essence the pretraining task becomes pretty standard cloze-question answering

This seems an obvious approach for cloze-type questions, but it seems non-obvious how to extent beyond this.

Are you aware of any work probing the differences in the representation using this style of masking vs a more normal language model objective? It would seem to me that this is the key to significant progress here (and of course one would speculate that a representation that works well for this would also work well for all kinds of KB-related tasks).

Thinking about this for a few minutes things like masking names, colors and numbers (the things that neural representations often confuse) and then asking questions based on them might be interesting. I wonder if bAbI could be extended for this?

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#68

Post by mjburgess »

hackinthebochs wrote: post #66
mjburgess wrote: post #65

Earlier quoted context omitted.

Who prepares the dataset? You are just shifting relevance problem to the human to solve. That's my point. The "regularity" isn't very hard to find when the data which exposes it is already selected.

No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorith…

identifying causes cannot be done statistically, as a fact of statistics

events A then B then C do not imply C is caused by B is caused by A

This problem is worse the more data you have, as with my example above of giving the machine /every/ event within a 1km radius of a pot boiling.

Identifying a cause is a dynamic experimental process. A machine can only accept a highly prepared dataset which has already been chosen because the associations are known to be causal.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#69

Post by hackinthebochs »

mjburgess wrote: post #68
hackinthebochs wrote: post #66

Earlier quoted context omitted.

No, I don't see your point. No one "prepared" common crawl to contain multiple instances of relevant examples for GPT-3 to learn from. It's just that a sufficiently large training corpus will naturally have this sort of regularity in it. My point is that a general learning algorithm trained on a sufficiently large and representative corpus will capture causal regularity without the sort of fine-tuning of the algorith…

identifying causes cannot be done statistically, as a fact of statistics events A then B then C do not imply C is caused by B is caused by A This problem is worse the more data you have, as with my example above of giving the machine /every/ event within a 1km radius of a pot boiling. Identifying a cause is a dynamic experimental process. A machine can only accept a highly prepared dataset which has already been chos…

But this problem isn't specific to machines. The fact that my hand hurts after touching an open flame stove doesn't entail the flame caused my pain. No amount of first-hand experience with flames can entail that flames cause pain. All we can do is increase the likelihood of this model. Experimental processes are in the same boat, except that the statistical power is greater. But our lived experiences with flames and pain and all other stimuli is a sort of low power ongoing experiment. But with enough of these poor experimental runs we can converge on an approximately true model. The same goes for a statistical learning algorithm and a large training corpus.

Re: Natural language benchmarks don’t measure AI models’ general knowledge well

#70

Post by mjburgess »

hackinthebochs wrote: post #69
mjburgess wrote: post #68

Earlier quoted context omitted.

identifying causes cannot be done statistically, as a fact of statistics events A then B then C do not imply C is caused by B is caused by A This problem is worse the more data you have, as with my example above of giving the machine /every/ event within a 1km radius of a pot boiling. Identifying a cause is a dynamic experimental process. A machine can only accept a highly prepared dataset which has already been chos…

But this problem isn't specific to machines. The fact that my hand hurts after touching an open flame stove doesn't entail the flame caused my pain. No amount of first-hand experience with flames can entail that flames cause pain. All we can do is increase the likelihood of this model. Experimental processes are in the same boat, except that the statistical power is greater. But our lived experiences with flames and…

Yes but the body solves the problem of concluding that the stove caused the pain.

The body is a mechanism of relevance. The more data you feed a machine, the worse it performs.

The body isnt necessarily right about causation (though it often is) but it provides a minimal mechanism of relevance to enable inference.

Data itself contains no causal information.

Post reply on HN