Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

21–30 of 313 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#21

Post by Systemerror7A69 »

To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well.

The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into it - but at least I would still call it a "next token predictor"

It's not using just training data, but what it's doing is predicting the next token to get to the solution. As far as my amateur knowledge goes, LLMs still roughly go token by token, deciding which one fits best given the context.

It's just not predicting based on it's training data, but predicting based on RLVR & more, trying to get to the optimal solution ( as much as the solutions CAN be optimal)

And I honestly think keeping this very much in mind is helpful in understanding and dealing with LLMs.

Re: “Next-token predictor” is the wrong mental model for LLMs

#22

Post by doc_ick »

mort96 wrote: post #20
atleastoptimal wrote: post #15

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.

100%

Re: “Next-token predictor” is the wrong mental model for LLMs

#23

Post by weego »

atleastoptimal wrote: post #15

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

imply that there is some fundamental limit on their capacities

This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI?

That seems more outrageous a claim than the one you're dismissing.

Re: “Next-token predictor” is the wrong mental model for LLMs

#24

Post by hirvi74 »

Sprotch wrote: post #6

I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?

My poor understanding is that an LLM does not "know" either. It basically uses probabilities to predict the next word based on a large matrix of probable outcomes.

For example, say I ask an LLM, "What sentence in English contains every letter in the alphabet?"

It would respond with something like:

"The quick fox jumps over the lazy, brown [next word]"

(Assume all the words were previously guessed correctly at this point)

The LLM guesses the last word based on what it has been trained on. Let's pretend the matrix is small, and the options narrow down to something like:

1. Dog (99.9% confidence) 2. Cow (85% confidence) 3. Bag (75% confidence) 4. Crayon (25% confidence)

The machine can confidently determine the final word of the sentence, "The quick fox jumps over the lazy, brown dog" because that sentence is unique because it is often used for testing things like fonts, a fun piece of trivia, and so on.

Brown Cow is not a bad guess because it's a type of cow and a yogurt brand. Brown bags and brown crayons are also perfectly rational adjectives to describe those common items and are not a bad guess either.

However, in the context of that sentence, dog is the most correct answer because one is unlikely to have written "The quick fox jumps over the lazy, brown crayon," thus it is quite improbable to be the answer.

My understand is this is where hallucinations can often come from. If the trivia about the sentence happened to not be in common in the data set, then "brown cow" might not be a terrible guess. There is clearly something rational behind that answer, but it's not correct in the sense that it answers the question correctly nor followed the instruction properly.

I'm sure the LLMs we have are far more capable these days. In fact, it wouldn't surprise me if an LLM could check its answer by counting the distinct letters in each word to verify. Not sure though.

Again, this is just a poor example based on my understanding, but I hope it helps (and is more correct than not).

Edit: Pretend word = token. It's technically tokens and not entire words, but I didn't not want to get into tokenization of words.

Re: “Next-token predictor” is the wrong mental model for LLMs

#25

Post by pjerem »

atleastoptimal wrote: post #15

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

Good example.

It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this.

The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical state, my senses…).

FWIW, I don’t believe LLMs are sentient, but I don’t think either that we have enough knowledge to rule it out.

Re: “Next-token predictor” is the wrong mental model for LLMs

#26

Post by epistasis »

Sprotch wrote: post #6

I understand how a computer can know that a chess move is more likely to lead to a win, and therefore “correct”, but I don’t understand how it can know that a token is correct. Can someone explain?

The LLM produces a probability distribution over the likelihood of all possible next tokens. So whatever the tokens are, "ch", "ex", etc. the next one gets a probability.

During training, real life text is fed through the LLM, and rhe "correct" token is the one actually observed in the training text. Here's a recent video walkthrough in some detail, mostly aimed at providing a deeper understanding than "next token predictor function":

https://youtu.be/GlYgs6v2YfU?is=IxVMhoCCE4N4WRVK

(Start at 15:30 for the LLM specific parts)

Re: “Next-token predictor” is the wrong mental model for LLMs

#27

Post by bluegatty »

'next token predictor' is a limited mental model but it's actually much better than any others.

'pattern matching' is a better intuition that 'reasoning' even though I think nominally, using the term 'reasoning' is perfectly fine in that context. It's just a loaded word that brings too much to the table.

'It hasn't seen the pattern' is a better description of the limitations of AI because it really just doesn't generalize very well at all. The adaptations described in the article don't change that.

Those are mutations, not expansions of capability.

Re: “Next-token predictor” is the wrong mental model for LLMs

#28

Post by saghm »

Systemerror7A69 wrote: post #21

To be honest, I believe I get the point the article is trying to make, and to an extent I agree, but I also think the point is not really made very well. The core of the argument as I understood it is that LLMs aren't just using existing data is training but also new ones. That's fine and good, and you can't simply assume an LLM is simply mashing together all it's data to give you an average of all that got fed into…

Yeah, it sounds like this is just a disagreement about what the word "next" means. I agree with you that "next" just means "the one about to come", and if the underlying model works by using some prediction mechanism to determine that, then it's by definition a next-token predictor. Disagreeing with that on the basis that the "next" token isn't necessarily in the training data verbatim just seems like an overly strict definition of the word "next".

Re: “Next-token predictor” is the wrong mental model for LLMs

#29

Post by gjm11 »

mort96 wrote: post #20
atleastoptimal wrote: post #15

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.

It is. And human beings are bags of chemicals. But for many purposes you will not find it helpful to think of human beings as bags of chemicals, and for many purposes you will not find it helpful to think of LLMs as next-token predictors.

Re: “Next-token predictor” is the wrong mental model for LLMs

#30

Post by atleastoptimal »

weego wrote: post #23
atleastoptimal wrote: post #15

"Next-token predictor" is one of those phrases used most of the time with a motive to downplay the abilities and faculties of AI models. It is intended to trivialize LLM's and imply that there is some fundamental limit on their capacities. Relying on it as a mental model for what LLM's are minimizes the emergent properties of scaling. It's like imagining that unicellular life could never eventually evolve into comple…

imply that there is some fundamental limit on their capacities This is a wildly dismissive statement that does a lot of heavy lifting. Your assertion is that we just happened to hit on a methodology that has no limitations between being an encyclopedia with a novel human language interface and, I guess by implication, AGI? That seems more outrageous a claim than the one you're dismissing.

I don't think it's outrageous when many of the people who claimed it was a next-token predictor have been proven wrong repeatedly over the past 5 years. There were people years ago who claims AI could never answer questions like "what would happen to a ball on a table if I moved the table" correctly because its text-base world model could never intuit physics, or that it could never do math or code accurately.

When I say there is some issue with people claiming there is some fundamental limit on the capacities of LLM's, I don't mean to say "If you think that they don't have unlimited potential you are wrong", I mean "you can't use the architecture of the transformer to make a sweeping declaration of things LLM's can or cannot do without empirical evidence, because the empirical evidence has unearthed far more surprising revelations than a reductive theory has been able to"

Post reply on HN