Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

301–310 of 325 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#301

Post by chrisjj »

HarHarVeryFunny wrote: post #292
chrisjj wrote: post #290

Earlier quoted context omitted.

> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. Who cares? Still it is next-token prediction.

The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing. Calling the base model a next token predictor is accurate since it is literally making a prediction and being g…

> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model,

Regardless, the statement being challenged here is "still next-token prediction, then".

> and I think the answer is no - not only is it not technically correct

It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predictor" remains accurate.

Re: “Next-token predictor” is the wrong mental model for LLMs

#302

Post by HarHarVeryFunny »

chrisjj wrote: post #301
HarHarVeryFunny wrote: post #292

Earlier quoted context omitted.

The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing. Calling the base model a next token predictor is accurate since it is literally making a prediction and being g…

> The question being originally asked is whether "next-token predicton" is the right mental model for an RL-trained model, Regardless, the statement being challenged here is "still next-token prediction, then". > and I think the answer is no - not only is it not technically correct It is correct. RL simply adjusts weights - with no effect beyond an equivalent adjustment to the corpus itself. Hence "next-token predict…

Yeah - but what is it adjusting weights based on? It's not based on next token ....

And per the focus of this thread, regardless of how accurate it is, why do you find "next token predictor" to be the most useful mental model?

Re: “Next-token predictor” is the wrong mental model for LLMs

#303

Post by danielmarkbruce »

YeGoblynQueenne wrote: post #300
danielmarkbruce wrote: post #155

Earlier quoted context omitted.

It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move. So, it's not a next move predictor. It's a game result predictor.

Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?

Brush up :)

The policy is optimized to maximize the the total reward, defined as the sum of the reward at each step, discounted by some factor.

Re: “Next-token predictor” is the wrong mental model for LLMs

#304

Post by bluegatty »

YeGoblynQueenne wrote: post #298
bluegatty wrote: post #27

'next token predictor' is a limited mental model but it's actually much better than any others. 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. 'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't gener…

>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a su…

fair point, agreed

Re: “Next-token predictor” is the wrong mental model for LLMs

#305

Post by Dylan16807 »

MrScruff wrote: post #258
Dylan16807 wrote: post #237

Earlier quoted context omitted.

If you're assigning steering 70 of your 100 output points because it's what you think we should go with most of the time in this situation, I'm going to call that a prediction of how to steer.

The point was, if your internal model of the world makes a prediction of a negative outcome at some point in the future, and you optimise your individual actions to avoid that negative outcome, then wouldn’t it make sense to focus on the fact you’re building and optimizing towards an internal world model rather than the fact you’re executing your actions one at a time in series?

If you purged and rebuilt that works model every action, the consequences of doing so would be worth a good amount of focus.

And if you didn't make the final decision on the action yourself that would be worth mentioning.

Re: “Next-token predictor” is the wrong mental model for LLMs

#306

Post by doc_ick »

danielmarkbruce wrote: post #295
doc_ick wrote: post #284

Earlier quoted context omitted.

Please feel free to cite sources then, otherwise I see no relevancy from you.

Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124

Thank you for providing an arxiv!

An aside, I finally do appreciate single column format now, makes it easier to convert to epub.

Re: “Next-token predictor” is the wrong mental model for LLMs

#307

Post by angoragoats »

garrinm wrote: post #150
angoragoats wrote: post #143

Earlier quoted context omitted.

> It doesn't check a "prediction" against what was actually "next" in data Literally no one here is claiming that it does. This is one of the many flaws in the article.

It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.

My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.

Re: “Next-token predictor” is the wrong mental model for LLMs

#308

Post by grebc »

danielmarkbruce wrote: post #296
doc_ick wrote: post #240

Earlier quoted context omitted.

Just like how reading a math book doesn’t teach you math, why do they make us read anyway? (Sarcasm) if reading a blog post didn’t teach someone how a car works how come it “can” work for next token predictors

Mine was sarcasm. People who actually understand cars have built them. Until you build something, you don't understand it.

Now you’re claiming people don’t understand unless they build something.

Boy, oh boy, do you keep digging your logic hole that much deeper.

As mentioned earlier, Sam thanks you for your obfuscation efforts while his equity keeps going up. The swindle continues.

Re: “Next-token predictor” is the wrong mental model for LLMs

#309

Post by danielmarkbruce »

angoragoats wrote: post #307
garrinm wrote: post #150

Earlier quoted context omitted.

It does in pre training, but not in RL post training. And not at inference time. Reading over all these comments I get the feeling my mistake was not clearly delineating inference time and train time.

My point is that “it doesn’t check the accuracy of the prediction against the data” is a non-response, because no one calling it a “next-token predictor” is making the claim that it does do that or that they’re calling it a next-token predictor because it does that.

Many people are in fact claiming the thing you are saying they are not - even if you are not. The reason they are claiming it is that it was true at one point, and most intro courses/blog posts/videos still describe them that way and then hand wave some "other stuff at the end". You can even see a comment here that refers to the gpt-2 paper. LLMs were trained to predict the next token, produced a distribution to do so, were scored against their prediction v the truth, and the weights updated so that the probability distribution made it more likely to predict the truth from that sample next time. They were, in every sense of the word, a next token predictor.

They are no longer that thing due to post training. They simply aren't making a prediction, and they aren't even optimized for the next token. If I give a distribution of the heights of the population, I'm not giving a prediction either. Distributions don't imply predictions.

Why the desperation to hang onto the word "prediction"?

Re: “Next-token predictor” is the wrong mental model for LLMs

#310

Post by danielmarkbruce »

doc_ick wrote: post #306
danielmarkbruce wrote: post #295

Earlier quoted context omitted.

Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads. https://arxiv.org/pdf/2411.15124

Thank you for providing an arxiv! An aside, I finally do appreciate single column format now, makes it easier to convert to epub.

When you are done with the section on RLVR, consider whether the model is predicting tokens, or making moves. There is a reason the word "policy" is used in RL.
Post reply on HN