Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

101–110 of 328 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#101

Post by a3w »

The errors LLMs typically made for me were looking for "schmutz" as a jiddish word, got "schmuck" on my screen. Thought why the stupid mistake? The next-token predictor model perfectly explains it.

Or starting with "yes". And this early locking in was a total lie, in the discussion that became "yes, might appear that way, but totally no since reasons". So it should have written "No", topmost, but could not self-edit that.

But nice that this gives it a more nuanced view, I might have to update my priors.

Re: “Next-token predictor” is the wrong mental model for LLMs

#102

Post by godshatter »

I think the author is arguing against the idea of a next-token predictor as something that simply uses the weights in the neural net which record the probabilities of tokens following other tokens as a valid definition of what an LLM is. Essentially a massive extreme markov chain.

With reinforcement learning and probably attention and other tricks that affect the weights based on things that aren't strictly in the training data, so the argument goes, you can end up with tokens following strings of tokens that would not be possible to be output with the training data and original weights alone. So describing it as solely a next-token predictor is incorrect based on this framing of it.

But that's just my take on this, I'm still trying to wrap my head around it all.

Re: “Next-token predictor” is the wrong mental model for LLMs

#103

Post by hirvi74 »

infamia wrote: post #57
hirvi74 wrote: post #9

Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.

> I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke. People don't know exactly the words that they're going to say necessarily, but tend to start with a general concept of what they're trying to communicate and only then try to put together the words (sometimes out of order). LLMs do not begin with any sort of conc…

> LLMs do not begin with any sort of concept they're trying to express.

Why do the need to? Considering they are merely tools, I actually appreciate they do not do this. A calculator can compute far better than any human, but I appreciate that calculators are not capable of expressing anything about the computations I request. I want the answer, not a conversation.

> LLMs are simulations that attempt to reproduce what an average person might say while wired up to a huge knowledgebase.

If you will allow me to be simplistic, people -- the soul, the self -- are predominately the aggregated effects of memories and experiences and the ability to retain new memories based on new experiences, no? Consider medical conditions in the dementia family of diseases. As memories fade into the ether, what remains of the self?

Also, people simulate/emulate each other all the time based on what an average, reasonable person might say. People incapable or unwilling to perform such mimicry are often labeled with all kinds of pejorative terms.

Re: “Next-token predictor” is the wrong mental model for LLMs

#104

Post by ordersofmag »

danielmarkbruce wrote: post #51
stanleykm wrote: post #5

> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?

The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.

The LLM does not determine the next token. It generate odds for all of the tokens it knows as to their likelihood of being 'next'. It's up to the harness running the LLM (and in most cases the a temperature setting) to actually decide on a particular next token. I think it's more accurate to call the thing the LLM actually generates (an ensemble of probabilities) a 'prediction'. It might be accurate to say the harness decides on the next token based on the prediction from the LLM. The role of the LLM is much more akin to predicting your opponents move than deciding your own.

Re: “Next-token predictor” is the wrong mental model for LLMs

#105

Post by Planktonne »

mannykannot wrote: post #68
Aurornis wrote: post #50

Earlier quoted context omitted.

This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable. With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.

You are not wrong, but I think atleastoptimal's suggestion was that descriptions like "next-token predictor" are often used to imply that there's not much to see here, perhaps with an implied "obviously" in there. "Stochastic parrot" seems to be a case in point: no-one expects intelligent, informed conversation or working software from a parrot - not even the eternally-resting Alex.

I don't that's a fair description of either 'next-token predictor' or 'stochastic parrot'. Both of those terms describe mechanism, not value--the fact that people squawk that the terms are minimising is projection on their part, not inherent to the phrase.

Re: “Next-token predictor” is the wrong mental model for LLMs

#106

Post by Kranar »

Dlemlo wrote: post #77
Systemerror7A69 wrote: post #21

To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into…

Blog articles from Anthropic and others show that this is not true. A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co. There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.

You're misinterpreting these articles.

Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed.

So "emits the next token" and "has no representation of anything beyond the next token" are two different claims. What autoregressive LLMs cost as a consequence of strictly outputting the next token is commitment. Once a token is output there's no going back. There's no revision or means of correction, and sometimes this can lead an LLM to route around its own earlier mistakes or simply produce false statements/hallucinations instead of going back and fixing them.

Re: “Next-token predictor” is the wrong mental model for LLMs

#107

Post by mofeien »

Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong:

1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector.

2. What you see when you use an LLM is not next-token prediction directly next to the prompt, but instead following a block of varying length of next-token prediction that happened to make progress on the problem in your prompt, and which just summarizes the results.

Re: “Next-token predictor” is the wrong mental model for LLMs

#108

Post by HarHarVeryFunny »

chrisjj wrote: post #10

Better title: Continue thinking of LLMs as Next-Token Predictors Because no, post training doesn't change that.

I'm not sure that's a useful way to think of it.

RL post-training changes the nature of what is being predicted, basically turning it from a copying machine into a goal-seeking machine.

A base model is predicting training sample continuations (copying).

A post-trained model is now steering/narrowing the base model's predictions in directions that were reinforced by RL goals.

The model is no longer predicting what the next token will be, but rather predicting what it should be in order to steer generation in the reinforced directions.

Re: “Next-token predictor” is the wrong mental model for LLMs

#109

Post by WarmWash »

Kranar wrote: post #106
Dlemlo wrote: post #77

Earlier quoted context omitted.

Blog articles from Anthropic and others show that this is not true. A LLM already knows more tokens than the current one. It was mentioned in a blog post about how a LLM is doing haikus and co. There are also structures in an LLM which allows it to 'estimate' numbers to a certain degree and doing other things.

You're misinterpreting these articles. Autoregressive LLMs generate tokens one at a time, disputing this is just plain wrong. What is true, however, is that in order to generate the next token autoregressive LLMs produce internal/hidden state about future tokens far past the next token so that it's not like the entire machinery of the algorithm deprives itself of representing where the sentence/text is headed. So "em…

I think the snag is that people use "Next token predictor" as a dismissive statement. In reality it's about as functionally dismissive as "humans only communicate one word at a time".

Re: “Next-token predictor” is the wrong mental model for LLMs

#110

Post by mofeien »

bluegatty wrote: post #27

'next token predictor' is a limited mental model but it's actually much better than any others. 'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table. 'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't gener…

How about "outcome steering" as a mental model? During training it is optimized until it's really successful at producing code / terminal commands / words that make the compiler/computer/itself do something that ultimately completes a long time-horizon task that iswcurrently being trained.
Post reply on HN