Earlier quoted context omitted.
John Searle Octopus Test
I assume you're referring to this? https://www.aclweb.org/anthology/2020.acl-main.463.pdf Gotta say in my opinion it didn't age well for a paper published in 2020 (but pre GPT-3). In practice, we have an O that is pretty good at describing how to build weapons against bears despite having absolutely no idea what "weapon" or "bear" refers to. But also, I don't seem to understand how this relates to the discussion, so…
Agreed. The glaring issue with their argument falls out of their own introduction. What is the "linguistic intent" of the sentence "When was Malala Yousafzai born"? Well, it falls right out of a natural language training corpus which will, with high likelihood, be followed up with her actual birthdate! So there is some non-trivial signal within the training corpus of linguistic intent. Capturing this intent is then just a computational problem. But the objective function of predicting the subsequent word has within the space of solutions the linguistic intent of the writers of the training corpus.