This is an incomplete understanding of what very large LMs are doing. At a very large scale new behaviors emerge[1][2]. It's true that the fluency of language models is easily explained by "predict next token given context" but that doesn't preclude the fact the LLMs are functionally doing reasoning up to some limits. To quote:
> However, it is unclear how these models obtain the answers and whether they rely on simple heuristics rather than the generated chain-of-thought. To enable systematic exploration of the reasoning ability of LLMs, we present a new synthetic question-answering dataset called PrOntoQA, where each example is generated from a synthetic world model represented in first-order logic. This allows us to parse the generated chain-of-thought into symbolic proofs for formal analysis. Our analysis on InstructGPT and GPT-3 shows that LLMs are quite capable of making correct individual deduction steps, and so are generally capable of reasoning, even in fictional contexts. However, they have difficulty with proof planning: When multiple valid deduction steps are available, they are not able to systematically explore the different options.
from "Language Models Can (kind of) Reason: A Systematic Formal Analysis of Chain-of-Thought"[3]
To summarise that paper, they create imaginary scenarios and get the LLM to answer questions. For example:
> Q: Every vumpus is a numpus. Each vumpus is dull. Dumpuses are vumpuses.
Every dumpus is not hot. Every impus is a dumpus. Impuses are brown.
Wumpuses are impuses. Wumpuses are opaque. Tumpuses are wumpuses. Tumpuses
are small. Every yumpus is hot. Zumpuses are tumpuses. Zumpuses are feisty.
Rompuses are zumpuses. Every rompus is not kind. Each jompus is a rompus.
Jompuses are sour. Alex is a zumpus. True or false: Alex is hot.
A:
(Appendix A.3)
This is a "5-hop" reasoning question, which the LLM fails at. But at 3 hop InstructGTP (davinci-002) scores 75% correct for completely fictional ontologies.
[1] https://arxiv.org/pdf/2206.07682.pdf
[2] https://hai.stanford.edu/news/examining-emergent-abilities-l...
[3] https://openreview.net/forum?id=qFVVBzXxR2V