Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

81–90 of 324 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#81

Earlier quoted context omitted.

Nope. Post training means even the raw model isn't predicting.

In what sense is the model not predicting?

There isn't a truth to test against. If I predict the next word in a sequence is "sat", we can check against the sequence. If I predict the roll of a die will be 4, we can check against it. Whether i give 100% or give a probabilistic prediction, we can check against the truth.

If I choose a specific move in chess, it's a choice. It's not a prediction. I might get a score 40 moves later given my choice, but I'm not predicting the next move.

To compare - during pre-training, the model literally tries to predict the next token (probabilistically), the training loop checks against the "right" answer, and the weights are updated based on that check. It's optimized to predict the next token.

Re: “Next-token predictor” is the wrong mental model for LLMs

#82

> Calling the second system a “next-move predictor” would be strange. It is not trying to predict what move appeared next in a dataset. It is trying to choose a move that wins. i dont understand the distinction here. does working backwards from a set of win states instead of working forwards from the current state somehow change whether it’s a prediction or not?

The word "predict" has a meaning. I don't "predict" my next move in chess. I might predict what someone elses first move is.

[deleted]

Re: “Next-token predictor” is the wrong mental model for LLMs

#83
post #63

I don’t think next token prediction is a particularly good description of pretraining either. The intermediate representations at each position are being optimised not only to help predict the next token, but also to help predict all subsequent tokens within the training context. You can see this directly in backpropagation: the gradient reaching a representation at position i sums contributions from prediction losse…

[deleted]

Re: “Next-token predictor” is the wrong mental model for LLMs

#85

Earlier quoted context omitted.

During training, certain tokens are more likely to lead to a lower loss function value, which is how you "win" the game of LLM output.

So, next-token predictors

And any next-comment predictor could have predicted your totally unhelpful, uninsightful, and unoriginal comment.

You -- along with everyone else who keeps parroting this thought-stopping phrase and other tired cliches like "stochastic parrot", simply because you heard other people say them, without understanding what they really mean, which published research papers they came from, or what those and other papers actually argued -- are desperately clinging to a reductive, short-sighted, shallow, simplistic model like a drowning person clutching a concrete life preserver.

Seriously, we are trying to throw you a lifeline, and you are refusing even to participate in your own rescue. So squawk for yourself.

https://news.ycombinator.com/item?id=48395727

> The term "stochastic parrot" is a slogan masquerading as an explanation, only a shallow surface description of the mechanism, that totally fails to explain the phenomenon, or account for all that LLMs and language itself can do.

Here is the original 2021 paper that coined the phrase. It was not primarily an argument about consciousness, nor did its title constitute experimental proof that everything an LLM does can be explained as parroting. It was principally a position paper about the risks of increasingly large language models: environmental and financial costs, biases and hegemonic viewpoints inherited from poorly documented training data, unequal access and power, and the danger of people attributing meaning and accountability to synthetic text.

The paper did, however, make a strong theoretical claim: because an LM is trained on linguistic form without direct access to communicative intent, it cannot possess meaning, understanding, or a model of the world. The authors described it as "haphazardly stitching together sequences of linguistic forms" according to statistical regularities -- hence "a stochastic parrot."

That distinction matters. The popular slogan discards the paper's detailed analysis of actual risks while treating its most controversial theoretical premise as an established scientific result. It has escaped into pop culture as a drive-by anti-LLM slogan -- something people repeat instead of investigating what these systems represent internally, how post-training changes their behavior, or what they can actually do.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, "On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?"

https://s10251.pcdn.co/pdf/2021-bender-parrots.pdf

Ironically, the objection that rhetoric was replacing scientific analysis appeared almost as soon as the phrase itself -- indeed, before the paper's formal publication. In January 2021, Michael Lissack published a response characterizing the draft as an advocacy piece that enumerated harms while leaving its assumptions, potential benefits, and cost-benefit trade-offs largely unexamined.

"The Slodderwetenschap (Sloppy Science) of Stochastic Parrots -- A Plea for Science to NOT Take the Route Advocated by Gebru and Bender"

https://arxiv.org/abs/2101.10098

Re: “Next-token predictor” is the wrong mental model for LLMs

#86

Text renderer, whatever. Doesn't matter how you think of them, they are very interesting technology that is being misused and misconstrued in the name of something that has nothing to do with technology: political economy. The greatest trick the rich ever pulled was making us think that the economy is about technology, and not politics.

Sure, politics and economics are involved - but why can’t technology also play an important role?

None of this AI political economy stuff was relevant in 2015 because necessary technological breakthroughs had not yet been made.

Re: “Next-token predictor” is the wrong mental model for LLMs

#87

Earlier quoted context omitted.

But it is a next token predictor. Recursively invoked. With carefully selected context. And massive investment in RL to tune token selection. And the ability to use cli tools on other folks' machines. That's a powerful system built around a conceptually simple technology: Next token predictors.

Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.

From another point of view, campaigning against the "next token predictor model" is a means to implicitly inflate LLMs' abilities. Given all the other hype-inducing terminology we've seen--"reasoning", most egregiously IMO--this seems more likely. Is there a simple, more accurate mental model? From what I've seen of the literature, "next token predictor" is a very accurate first order description of what an LLM does, I can't really do better, therefore this or that connotative interpretation isn't giving me a great deal of pause.

Re: “Next-token predictor” is the wrong mental model for LLMs

#88
post #39
post #9

Sure, I get the gist of the article. I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. By that rational, the human brain is really not that much different. When I am having a conversation with another person, I do not usually have every word I will respond with stored in my limited working memory. My output is often predicted based on the previous word I spoke.

> I have never liked the reductionist argument that LLMs are nothing more than next-token predictors. I have never heard such an argument. Recognition that LLMs are nothing more than next-token predictors does not come from reductionism. It comes from simply knowing how they work e.g. from viewing the inference code.

J.S. Bach said something similar about music and keyboard instruments.

> "There's nothing remarkable about it. All one has to do is hit the right keys at the right time and the instrument plays itself."

My issue is not with fact at face value. My issue is with how the fact is often contextually used in arguments to delegitimize and disparage LLM outputs and LLM users.

Yes, LLMs at a fundamental level are next-token predictors. But in my opinion, LLMs are very useful, imperfect next-token predictors.

There are a lot of wannabe John Henry [1] folks out there. Love LLMs or hate'em, most of those John Henry folks ain't beating these machines on a plethora of tasks.

[1] For those unaware, https://en.wikipedia.org/wiki/John_Henry_(folklore)

Re: “Next-token predictor” is the wrong mental model for LLMs

#90

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

> used most of the time with a motive to downplay the abilities and faculties of AI models

Exactly. We're dancing around the real argument: there's massive amounts of influencing going on (and not only about AI.)

Post reply on HN