Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

161–170 of 330 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#161

Post by danielmarkbruce »

zahlman wrote: post #115
mofeien wrote: post #107

Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong: 1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector. 2. What you see when you use an LL…

Even so, one might wonder why we don't try making systems that take different approaches. For example, after a traditional first pass of output, they could do sliding-window "optimizations" considering each token in the context of tokens both before and after , and possibly replace words or phrases in-place. For example, I've noticed quite a few cases recently of LLMs outputting "but" where "and" would make more sens…

People have and are trying things. Lots and lots of things. They just don't go around promoting failed ideas.

Re: “Next-token predictor” is the wrong mental model for LLMs

#162

Post by danielmarkbruce »

saghm wrote: post #129
jameshart wrote: post #97

Earlier quoted context omitted.

But if you are an NFL coach and I ask you to decide your next action in order to maximize your odds of winning the superbowl , while yes that does involve you having some predictive ability to think about what impact your actions would have on your odds of winning the Super Bowl… I don’t think you would call the process that you use to decide that next action ‘prediction’.

I don't find the fact that I don't call any humans "action predictors" to be a particularly meaningful insight because my rationale is that it's a weird thing to call a human; football coaches can do plenty of other things besides just coaching football.

Predict implies you don't control a situation. That's the difference.

Re: “Next-token predictor” is the wrong mental model for LLMs

#164

Post by wat10000 »

danielmarkbruce wrote: post #159
wat10000 wrote: post #147

Earlier quoted context omitted.

The discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all. The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of…

The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token. What is emitted by a model during RLHF and RLVR is not, by…

Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, most of the objections to "next-token predictor" are of the latter form.

Re: “Next-token predictor” is the wrong mental model for LLMs

#165

Post by danielmarkbruce »

grebc wrote: post #160
danielmarkbruce wrote: post #149

Earlier quoted context omitted.

If you haven't built one, and don't understand how they work, why comment?

You don't need to build a car to understand one. That you tie yourself up in knots of fancy acronyms instead of plain words and that your argument boils down to semantics of the word prediction, it's pretty clear what is up brother.

Lol, sure, just read a blog post and you'll understand how a car works....It's very simple....

Re: “Next-token predictor” is the wrong mental model for LLMs

#166

Post by [deleted] »

wat10000 wrote: post #147
gwerbin wrote: post #137

Earlier quoted context omitted.

But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.

The discourse around this is annoying. A big group of people use "next-token predictor" to imply that LLMs aren't capable of anything interesting. Another big group of people opposes the use of "next-token predictor" because of that implication. But that fight isn't about the "predictor" language at all. The linked article makes a good point: a substantial chunk of the training does not consist of "here's a bunch of…

[deleted]

Re: “Next-token predictor” is the wrong mental model for LLMs

#167

Post by danielmarkbruce »

wat10000 wrote: post #164
danielmarkbruce wrote: post #159

Earlier quoted context omitted.

The fight is about the predictor language in some cases. Because it's only a trivial difference to those who don't understand the details of how these things are made. In pre-training the model really is trained to predict the next token. What is being emitted by the model is, by structure, by training and by optimization, a prediction of the very next token. What is emitted by a model during RLHF and RLVR is not, by…

Right. But that's never what the discussion is about. This is the first time I've seen someone say, "it's not just a next-token predictor because a bunch of the training isn't about predicting the next token." I have seen many, many people say "it's not just a next-token predictor because there's all this smart stuff going on inside the model." Even here, in the comments for an article that's all about the former, mo…

Yup, you are mostly right.

I guess the people in my camp find the "it's just a next token predictor" stupid in that it's like saying "it's just a bunch of carbon and hydrogen", but it's also one of those things where people like to think they are clever because they think they are theoretically correct. But they aren't even that. So it's like double stupid. But the "next token predictor" part is at least technically correct (like, carbon and hydrogen right) for pretraining, so the debate can't really be won there.

Re: “Next-token predictor” is the wrong mental model for LLMs

#168

Post by Sprotch »

epistasis wrote: post #26
Sprotch wrote: post #6

I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?

The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability. During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token pr…

Thanks - that makes sense. On that basis the article’s thesis is totally wrong - it would be like a computer program rating its ability based on how well it predicts moves played by grandmasters in the past. It’s not inventing new moves.

Re: “Next-token predictor” is the wrong mental model for LLMs

#169

Post by dijksterhuis »

danielmarkbruce wrote: post #154
Kranar wrote: post #106

Earlier quoted context omitted.

You're misinterpreting these articles. Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed. So "em…

Emitting and predicting are different things though. Prediction implies there is some "truth" or event or something that you can test against. Prediction implies the model just learns from existing text, and optimizes to predict the next token in training data. That's just not true.

prediction is a very specific term of art in the field of machine learning. generally speaking, machine learning models like LLMs are based on probability; performing a statistical prediction of the likely y given some input x

    Probability(y | x)
that's why we refer to outputs as a prediction. it is likelihoods and stuff. the output is never definitely correct as we're not dealing with heuristic processes.

> Prediction implies there is some "truth" or event or something that you can test against

there absolutely is a ground truth during training. the core predict-the-next-most-likely-token part of an LLM has a ground truth next-token. that's why you don't end up with generated text like: fish spurious send cattle chocolate phone happy meaning ball orange board canada.

> optimizes to predict the next token in training data

that is the optimization goal in training the next-most-likely-token core of an LLM, it basically translates to maximise the likelihood of predicting the next token x_i given the previous tokens

    L(θ) = −log Π^n_{i=1} f_θ(x_i | x1, ..., x_{i−1})
https://arxiv.org/pdf/2012.07805 (GPT2 but the point still stands)

(edit: sorry for the ADHD edits)

Re: “Next-token predictor” is the wrong mental model for LLMs

#170

Post by garrinm »

Systemerror7A69 wrote: post #21

To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into…

I think the point is more that in RL there's no ground truth to predict. So when training a model with RL the idea of "predicting" doesn't fit anymore. I'll make some edits I see that I wasn't very clear.
Post reply on HN