- It is a next token predictor OP https://blog.florianherrengt.com/how-llms-work.html
- deal with it and stop whitewashing, greywashing it
“Next-token predictor” is the wrong mental model for LLMs
211–220 of 330 posts
Re: “Next-token predictor” is the wrong mental model for LLMs
#212Earlier quoted context omitted.
The distinction is that it's not 'predicting the next token'. Instead it's _determining_ the next token based on a prediction of its reward signal.
Yes, but I think the same construction could also be used to characterize the first system; it determines the next move based on a prediction of its reward signal, where its reward signal is a measure of how likely it is that a grand master would make that move. Like stanleykm, I found this analogy somewhat puzzling. On reflection, I think the author's point is this: the statistics of actual usage do not seem suffici…
A classically pretrained LLM does not have a concept of having determined its previous tokens — it has only ever observed inputs that it had no causal influence over. This is why it's valid to say its actions are predictive and not determinative.
Re: “Next-token predictor” is the wrong mental model for LLMs
#213Earlier quoted context omitted.
Yes, it was. Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. You aren't in this field. You are clearly wrong and just can't handle it.
> Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. To understand how an engine works, it's important to understand what a piston does as part of the engine.
You are conflating "half built" with "a piece of a system".
The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
Re: “Next-token predictor” is the wrong mental model for LLMs
#214Re: “Next-token predictor” is the wrong mental model for LLMs
#215To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into…
It is a bit of a pedantic argument but I get upset when people misuse the term, abstraction, and can feel the impulse to write a blog post like this.
I think it’s important to make clear the RL part and the alignment and parameter tweaking that gets done on models and inference engines. It makes them more understandable as mechanisms and less like wish-washy super intelligences that make their own decisions.
When these systems win math Olympiads, it’s not terribly surprising or interesting to me. Of course they will: we trained models to play nearly optimal chess games and Go. You tweak the rewards and sigmoid and you start optimizing the function towards your goal. This is how learning systems of all stripes work.
It’s still next-token predicting at the end of the day. I don’t think it’s a reduction-ad-absurdum.
But a lot of people still call it, “intelligence,” and try to use language that obscures what is happening in terms of anthropomorphic behaviour and not machine ones. That really does influence how we use these tools and profits those who would use them on us.
Re: “Next-token predictor” is the wrong mental model for LLMs
#216Earlier quoted context omitted.
The blog post itself says one thing, but then demonstrates the exact thing they’re arguing against. If you can’t grasp that logic gap then there’s no point discussing further.
I try to make 3 claims in the post, it was a bit clumsy I'll admit that. 1. At inference time, LLMs emit one token at a time given the prior tokens. This looks like prediction and I concede that. 2. During pre-training, LLMs predict the next token and compare to the actual next token in the training data. This is the classic setting for ML predictions. And I think its meaningful, the model really is predicting what t…
Probably the easiest way to describe an LLM that it's a policy. There is a reason that word has stuck in RL.
And it's not just RLVR. RLHF has been going on for years and years. LLMs have not been "next token predictors" for probably 5-6 years.
Re: “Next-token predictor” is the wrong mental model for LLMs
#217Earlier quoted context omitted.
You're misinterpreting these articles. Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed. So "em…
I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".
this is a great way of expressing it.
Re: “Next-token predictor” is the wrong mental model for LLMs
#218Earlier quoted context omitted.
> Nobody says a system or product works a certain way and means the system while it's half built. "Bridges drop cars in the water!". Right. To understand how an engine works, it's important to understand what a piston does as part of the engine.
You are conflating "half built" with "a piece of a system". The model weights change as the model goes through the training process. They aren't stored after pre-training is done and other weights are put somewhere else. It's more like pottery - the thing changes. It's not correct to say something is soft and malleable because it once was.
[deleted]
Re: “Next-token predictor” is the wrong mental model for LLMs
#219The "next-token predictor" is taking a design stance (see Dennett). But to take the intentional stance is more interesting in case of large language/reasoning models.
Re: “Next-token predictor” is the wrong mental model for LLMs
#220To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into…
'Prediction' gets overloaded with optimization. Predictions are binary, optimizations are fuzzy.
If you're saying it's predicting, then each result should be falsifiable.
The result of an LLM output should be able to be scored against what it is supposedly predicting. Of course, that isn't possible, because it isn't predicting anything when giving novel outputs, otherwise that thing would exist independently.