So, what does a sentence produced by GPT3 signify, and what does it depict? What it signifies (ie causal history) is that this sequence of words is what human writers would most likely use when producing an utterance containing whatever trigger words the model has been fed. In other words, it's a statistical modlel. What it depicts is whatever the mapping rules for that language tell us it depicts. Since human beings usually tell the truth, a statistical model will usually produce true statements. It will also seem to have the ability to tell true from false, in many cases. But because GPT3 has no model for the significance of its sentences, it cannot be said to have any understanding of language, in the sense humans have it. LeCun's point about flying an aeroplane to the moon is essentially correct.
Re: Yann LeCun on GPT-3
#221The relationship between language and the world is this: utterances both signify and depict objects and events in the real world. So, if I say "I saw Alec Baldwin at the bastketball game last night", then that depicts an event in the real world. And, if and only if the statement is true, an event similar to that depicted was part of the causal history of the utterance itself. The causal history of the utterance determines the significance of the utterance, just as the causal history of a footprint determines its' signficance. To understand a sentence is to understand what it depicts in the real world, and what it actually signifies in the real world. The ability to tell true from false is the ability to detect a disconnect between what is depicted and what is actually signified. That is what LeCun implicity means by language understanding.