“Next-token predictor” is the wrong mental model for LLMs

Yes, it was. Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right.

You aren't in this field. You are clearly wrong and just can't handle it.

Here's my take on the "next-token predictor" idea, from a much longer article I wrote recently:

https://www.oranlooney.com/post/rose-petals/#language-models

It’s popular to dismiss LLMs as “just next token predictors.” This is technically true, but also kind of misses the point. Markov chains, RNNs, and transformers are all language models that can be described as “next token predictors,” but they don’t all work equally well. A better question to ask is: “What is this model’s inductive bias?”

A Markov chain (an -gram model) assumes the next word depends on the previous words, and that each possible combination of words has a completely independent parameter. (Andrey Markov proposed using this language model over a century ago, making it the granddaddy of modern LLMs.) So, for a vocabulary of size , there are parameters to learn. For even a smallish like 5, that already explodes the hypothesis space beyond what can be learned from even a huge text corpus like the entire internet. And, simultaneously, having a context window of only the previous 5 words is grossly inadequate for modeling real-world language. Like our FCNN above, this model suffers from having an inductive bias which is too weak.

RNNs tried to fix this problem by compressing the entire history into a single fixed-size state vector, updated one token at a time. But that compression is itself a brutal assumption: everything worth remembering about the past must survive being squeezed through a tiny bottleneck at every step. In practice, RNN models quickly lose the plot after a handful of sentences. Locally, the text they generate looks grammatically correct and meaningful, but zoom out a little and they’re basically nonsense generators. Like our naïve linear model, this model suffers from having an inductive bias which is too strong.

Transformers manage to hit a sweet spot: by keeping the recent history around as a working memory, and attending to different parts of it at different times, the transformer’s bias matches real structure in language: the referent of a pronoun, the subject of a verb, the parenthesis waiting to be closed. Not only that, but the particular structure of the transformer, basically a weighted sum of semantic vectors from the context window, has empirically been shown to somehow be a “good enough” match for the structure of real-world language found in the wild.

Transformers aren’t “smarter” than other possible language models, they just happen to land in that Goldilocks zone where their inductive bias is just right.

Diffusion LMs denoise a canvas which I personally find more interesting.

I don't really disagree that human cognition is essentially a predictive task though, as I understand it, predictive coding and related theories based on the Bayesian brain hypothesis are fairly popular these days (though maybe not clearly dominant over alterative models? IDK I'm not a neuroscientist). I imagine most people would draft a few tokens before refining them like MTP or diffusion though, if we do decide to use LMs as an analogy to human cognition.

I feel like RLHF has a pretty obvious ground truth, human feedback is used as an (albeit noisy) signal of average human preferences. Same thing with RLVR and "solving the problem".

No, those aren't synonyms at all.

If I steer a car to avoid a predicted collision with a wall, this is not me 'predicting' the car. I am steering the car based on a prediction.