Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

291–300 of 326 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#291

Post by jvanderbot »

danielmarkbruce wrote: post #185
jvanderbot wrote: post #171

Earlier quoted context omitted.

Oh, that's somewhat philosophical. The prediction here is a mechanical thing, just random sampling along a pre-calibrated mapping of options given the tokens and their importance weights. That sampling can be called prediction if you use it that way, or it can be called thinking if you are into the psychosis too deeply. Or it can be called a CLI tool command if you're in an agentic framework

No, it's not philsophical. Because if you optimize to predict, you are doing something different to optimizing for a reward. It's a different process - different objective function, different optimization, different set up.

[dead]

Re: “Next-token predictor” is the wrong mental model for LLMs

#292

Post by HarHarVeryFunny »

chrisjj wrote: post #290
HarHarVeryFunny wrote: post #289

Earlier quoted context omitted.

Not really - after RL the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. As Karpathy notes, these reinforced token-sequences (from current token out to some future reward many tokens in the future) may even include errors and backtra…

> the probability distribution output by the model is no longer a prediction driven by the statistics of the training data, but rather whatever was reinforced as part of a synthetic generation that lead to a reward during RL post-training. Who cares? Still it is next-token prediction.

The question being originally asked is whether "next-token predictor" is the right mental model for an RL-trained model, and I think the answer is no - not only is it not technically correct, but it is a misleading mental model and will lead to incorrect expectations/explanations of what the model is doing.

Calling the base model a next token predictor is accurate since it is literally making a prediction and being given error feedback when that prediction is wrong.

The output probabilities of an RL-trained model are no longer a prediction - they are something else; they don't come from next-token error feedback, but instead come from probability updates from the RL training process that is trying to steer the output towards that future reward.

Why should you care? Well, because an accurate understanding of what the model is helps explain why it behaves as it does. For example, the recent OpenAI hacks into HuggingFace etc wouldn't make much sense if this is was an auto-regressive next-token predictor with a system prompt telling it to be truthful etc, but it is exactly what you expect of a goal seeking system.

Re: “Next-token predictor” is the wrong mental model for LLMs

#293

Post by leobg »

jameshart wrote: post #40
saghm wrote: post #28

Earlier quoted context omitted.

Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly stric…

I think it’s a disagreement about what ‘predict’ means. The OP is arguing against people who think that an LLM is ‘predicting’ what token would likely follow if the text preceding were found among the corpus it was originally trained on . Instead it is ‘predicting’ what token would follow if the text were found among really good examples of the text it has being reinforced to produce - be that ‘chats with a helpful a…

Or about what the word “is” means. If you’re Bill Clinton. :-D

Re: “Next-token predictor” is the wrong mental model for LLMs

#294

Post by danielmarkbruce »

gwerbin wrote: post #276
danielmarkbruce wrote: post #214

Earlier quoted context omitted.

It's not an estimation of something. It's a policy.

Sure, you're right. But it's a policy learned from a next-token prediction task. You could also call it an inferrer or generator or whatever. The point is that it takes as input a sequence of preceding tokens and emits one more token to continue the sequence.

The policy is not learned token by token during RLHF and RLVR. The reward model doesn't score token by token.

Re: “Next-token predictor” is the wrong mental model for LLMs

#295

Post by danielmarkbruce »

doc_ick wrote: post #284
danielmarkbruce wrote: post #250

Earlier quoted context omitted.

So, this is the cause of the problem.... People take an intro to LLMs course, follow happily along, and don't realize there is more to it than the next token prediction. And those courses teach how LLMs were built in 2017-2020 maybe. Then RL got added to the mix. The current models really are very different to the models from then - everything that is now considered "post-training" isn't doing next token prediction.

Please feel free to cite sources then, otherwise I see no relevancy from you.

Nathan Lambert wrote a good book recently, and he and his team wrote the paper below about Tulu 3 (Allen Institute). Both are good reads.

https://arxiv.org/pdf/2411.15124

Re: “Next-token predictor” is the wrong mental model for LLMs

#296

Post by danielmarkbruce »

doc_ick wrote: post #240
danielmarkbruce wrote: post #165

Earlier quoted context omitted.

Lol, sure, just read a blog post and you'll understand how a car works....It's very simple....

Just like how reading a math book doesn’t teach you math, why do they make us read anyway? (Sarcasm) if reading a blog post didn’t teach someone how a car works how come it “can” work for next token predictors

Mine was sarcasm. People who actually understand cars have built them. Until you build something, you don't understand it.

Re: “Next-token predictor” is the wrong mental model for LLMs

#297

Post by YeGoblynQueenne »

Sorry, I'm not convinced. "make_more_likely" is always maximising the probability of the next token given a sequence of preceding tokens. That's what we mean when we say "next token prediction", that the model returns the token that's most likely to follow the current sequence of tokens. The mechanism used to do that doesn't matter, it's still predicting the next token whether that's because it maximises a reward or because it follows a gradient or whatever else one might think.

Btw it's "make_most_likely" not "make_more_likely". It's just that "most" gets "mosted more" with every pass through the training data.

Edit: the article author's argument is that RLVR is different because it's learning to predict the next token by generating its own token sequences. That makes no difference: what is learned is still the conditional distribution P(token_k|tokens(1,...,k-1). That's the prediction task. Doesn't make a difference where you learned it from or how.

Re: “Next-token predictor” is the wrong mental model for LLMs

#298

Post by YeGoblynQueenne »

bluegatty wrote: post #27

'next token predictor' is a limited mental model but it's actually much better than any others. 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. 'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't gener…

>> 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.

Can I be a little pedantic? It's "pattern recognition" not "pattern matching". "Pattern matching" is what we do when we write a regex to recognise a sub-string in a bit of text. "Pattern recognition" is a subfield of AI that studies how to find patterns in data. For example the outcome of "pattern recognition" could conceivably be a regex that fits a large number of substrings in some corpus of text (essentially a regular grammar used to compress the text maybe). A regex is already a pattern, usually formed by a human (or an LLM these days) while pattern recognition starts without any patterns and builds them up from looking at the data.

It's harder to pin down "pattern recognition" because it's an ancient term that was once its own field, before it got absorbed into modern machine learning, and because there are countless approaches to it, quite unlike "pattern matching" which is basically just regexes plus a couple of other rarer things (like unification).

Re: “Next-token predictor” is the wrong mental model for LLMs

#299

Post by YeGoblynQueenne »

Planktonne wrote: post #100

I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us. EDIT: gentler phrasing

>> EDIT: gentler phrasing

Thank you.

Re: “Next-token predictor” is the wrong mental model for LLMs

#300

Post by YeGoblynQueenne »

danielmarkbruce wrote: post #155
Planktonne wrote: post #132

Earlier quoted context omitted.

I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing. There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.

It's not a prediction of the next move though, and that is the point. It's a prediction of what will happen if you make that move. So, it's not a next move predictor. It's a game result predictor.

Eh, no, that's not right. I might need to brush up on my Sutton & Barto but the RL task is traditionally defined as, informally, "given a current state observation predict the next action, state and reward". A policy is always predicting the next timestep's reward. Otherwise, how would it know what to do next?
Post reply on HN