Earlier quoted context omitted.
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
You cannot put classical particles anywhere.
[dead]
“Next-token predictor” is the wrong mental model for LLMs
61–70 of 313 posts
Re: “Next-token predictor” is the wrong mental model for LLMs
#62> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move.
Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem sufficient to produce a fluent LLM; it also takes reinforcement learning.
Re: “Next-token predictor” is the wrong mental model for LLMs
#63I don’t think next token prediction is a particularly good description of pretraining either.
The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context.
You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losses for subsequent tokens, not just from the loss for token i+1.
Re: “Next-token predictor” is the wrong mental model for LLMs
#64Earlier quoted context omitted.
>I don’t understand how it can know that a token is correct. It can't. The next token is just the most statistically probably given the context (at least in transformers). Try a very small/weak model in your own machine and more often than not it would get stuck repeating the same word or even just output garbage. Because after training and quantization (where some information is lost), that's the most probable next…
classical particles obey deterministic state evolution rules, yet put 10 in a box and you cannot say where they will be 5 minutes later.
Your point being? llms are software running in a fixed pipeline (barring variance induced by randomness in some layers). There's nothing like decoherence/thermal effects in a "lower level" that could induce stochastic behavior in a higher level. There's no Brownian motion in llms, if that's your analog with classical mechanics. You might argue emergent behavior that would look ordered/intelligent in some way and nature's full of examples of that but we don't attribute intelligence to physical processes.
Re: “Next-token predictor” is the wrong mental model for LLMs
#65A better statement might be: Current agentic systems may be *built* from next-token predictors which are conceptually simple, but because of agentic frameworks, recursive invocation, tool use, and *heavy* investment in reinforcement learning in these contexts and for specific applications, they can no longer be thought of as "Merely" next token predictors. Modern agentic work is probably more of a "emergent system fr…
Nope. Post training means even the raw model isn't predicting.
In what sense is the model not predicting?
Re: “Next-token predictor” is the wrong mental model for LLMs
#66> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
> _determining_ the next token based on a prediction
Also known as predicting.
Re: “Next-token predictor” is the wrong mental model for LLMs
#67Earlier quoted context omitted.
imply that there is some fundamental limit on their capacities This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI? That seems more outrageous a claim than the one you're dismissing.
I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately. When I…
> it could never do math or code accurately.
They still can't do code accurately. The fact that you use this as a defense of your position greatly undermines the credibility of your claim.
Re: “Next-token predictor” is the wrong mental model for LLMs
#68Earlier quoted context omitted.
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable. With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.
You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.
Re: “Next-token predictor” is the wrong mental model for LLMs
#69Earlier quoted context omitted.
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
> _determining_ the next token based on a prediction Also known as predicting.
I think the most useful word in both cases is "extrapolating".
An LLM extrapolates from its context window to the immediate next token. This word applies whether you view what's happening as "reasoning", "prediction", or as a math function.
Re: “Next-token predictor” is the wrong mental model for LLMs
#70Earlier quoted context omitted.
At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.
It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.
It can be pretty helpful to think of human function in chemical terms. Its at least unhelpful to deny it.