Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

“Next-token predictor” is the wrong mental model for LLMs

gmcgoldr.github.io

Post reply on HN
Print view View original

131–140 of 330 posts

Re: “Next-token predictor” is the wrong mental model for LLMs

#131

Post by mofeien »

Aurornis wrote: post #50
mort96 wrote: post #20

Earlier quoted context omitted.

At the same time, it ... is literally a next token predictor. Like that's what it is. The input is a sequence of tokens. The output is a probability distribution of next tokens.

This comment attracted a lot of analogies trying to reduce something to something else (calling humans a "bag of chemicals"), but the flaw in those analogies is that they're reducing something valuable to something that sounds less valuable. With an LLM, the tokens are the valuable part. That's what I want from it. That's why it exists. The tokens are the point, and it produces those tokens one by one for me.

Simple Markov chains are next token predictors, and they can provide you with much more tokens than you can consume, and much cheaper than from llms. Unbeatable in price and simplicity.

But there is no trillion dollar industry around cheap top Markov models. So there must be something about LLM tokens that makes them more valuable than those generated from a simple Markov chain. And that substance, that makes one valuable and the other not, is exactly what reduction to "next-token predictors" masks.

Re: “Next-token predictor” is the wrong mental model for LLMs

#132

Post by Planktonne »

garrinm wrote: post #122
Planktonne wrote: post #100

I'm not going to stop describing things accurately because someone generated an article that continually undermines its own main point. Limiting the way we talk and think about LLMs to a very narrow set of terms doesn't help us. EDIT: gentler phrasing

The distinction I perhaps didn’t make clearly enough is that I’m not really debating the concept of prediction at inference time, although, as I pointed out elsewhere, I think that’s the less interesting interpretation of what “prediction” means. What’s more interesting to me is its application at training time. In reinforcement learning, there is no ground-truth next token to predict. So if you’re comfortable callin…

I think you're trying to limit the meaning of both 'next' and 'prediction' in ways that don't reflect usage and that--if adopted--would severely limit our ability to discuss and evolve what LLMs are actually doing.

There's nothing inherent in either word that forces such a limit; predicting based on what will lead to success as measured by [reward function] is still a prediction.

Re: “Next-token predictor” is the wrong mental model for LLMs

#133

Post by garrinm »

Planktonne wrote: post #123
garrinm wrote: post #121

Earlier quoted context omitted.

It was written by a human. There are AI edits but it’s very much a human composition. Perhaps a bit sloppy.

In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.

I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.

Re: “Next-token predictor” is the wrong mental model for LLMs

#134

Post by Planktonne »

mofeien wrote: post #107

Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong: 1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector. 2. What you see when you use an LL…

> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step

I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words.

It's not minimising to describe even a complex prediction process as prediction.

Re: “Next-token predictor” is the wrong mental model for LLMs

#135

Post by astrobe_ »

deathanatos wrote: post #120
zahlman wrote: post #111

Earlier quoted context omitted.

> generated an article that continually undermines its own main point. I disagree that this accurately describes TFA.

The entire second on chess engines is, from the view of the entire thesis of TFA, is incoherent. Let's assume, for sake of argument, that I agree with the section: that an idealized chess move predictor isn't a predictor — which is not a thing that exists, as the space of chess is enormous, but let's pretend! — that's not what LLMs are? Even if we just restrict ourselves to the space of written English prose, the spa…

The reward function seems indeed to be the protagonist there, still it stays in the shadows. One can only imagine that it is some kind of evaluator that scores the sequence based on grammar correctness, semantic consistency, etc.? To use the proposed chess analogy, maybe it could be a Stockfish engine that evaluates the submitted position that results from the move submitted by the LLM?

Re: “Next-token predictor” is the wrong mental model for LLMs

#137

Post by gwerbin »

Planktonne wrote: post #134
mofeien wrote: post #107

Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step is doubly wrong: 1. In order to select even the first word of a meaningful sentence, it already has to have structure and meaning of what follows captured somewhere inside, mostly in it's weights/activations or indexed by it's state vector. 2. What you see when you use an LL…

> Describing it as a "next-token predictor" in the sense that this would mean it's fundamentally limited to just a fraction of an inferential step I don't think anyone is doing that though; we know LLMs are not simple Markov chains, and that the prediction they make is based on more than the previous X words. It's not minimising to describe even a complex prediction process as prediction.

But it literally is making a prediction based on the previous X tokens, it's just that X is huge and there is a proportionally huge number of parameters in the token generation function.

Re: “Next-token predictor” is the wrong mental model for LLMs

#138

Post by Planktonne »

garrinm wrote: post #133
Planktonne wrote: post #123

Earlier quoted context omitted.

In my experience, people who do 'AI-assisted' writing tend to be very bad at noticing how much of their work AI has changed. I'm sure you put thought into it, but passing it through AI takes a lot of that out.

I think that’s fair, I didn’t actually run the whole thing through an AI. it was more targeted edits, but each time it does erode at my writing. But at the same time, I don’t think it’s a good reason to dismiss this. Because I did spend several hours writing it, and I did put a lot of thought into it, and it was not in any meaningful way generated by AI.

Mostly I disagree with the article's ideas, if that helps; the AI was just a secondary factor.

> I don’t think it’s a good reason to dismiss this

AI-generated prose reads as sending a 'lack of effort' signal to a lot of people, just as no editing at all does. ; It's an effective heuristic that we've all learnt in the last couple of years.

In either case, it's not always fair: there are people who deeply care about their ideas but forget to fix basic errors, or pass it through AI.

In both cases though, the advice is the same: if you want people to take your output seriously, you need to signal that you are taking it seriously. That used to mean editing for spelling and grammar. Now it means not using AI.

Re: “Next-token predictor” is the wrong mental model for LLMs

#139

Post by zahlman »

otabdeveloper4 wrote: post #42
pjerem wrote: post #25

Earlier quoted context omitted.

Good example. It’s also like saying our brains are just electric circuitry incorporated in meat. It’s true but it seems that consciousness emerges from this. The fact that LLMs are next token predictors isn’t the interesting or impressive part. Actually my brain strictly is a black box predicting (or choosing) my next word/action/move… based on a complex existing context (my thoughts, the environment, my physical sta…

> It’s true It's not. "Brains as electrical circuits" is a gross simplification based on our ignorance and prejudices. (In the 18th century they spoke of brains as "clockwork mechanisms".) LLMs, in contrast, are literally next token predictors. We know exactly how LLMs work, and they are exactly that.

> We know exactly how LLMs work

I think you imply a rather loose standard for "exactly" here. I wouldn't even say this of major deterministic software projects that are orders of magnitude smaller than frontier LLM weight-dumps. In principle we could work our way through these systems eventually, sure, maybe even a single person could do so. But if we really understand exactly how our software works, how have we been tolerating bugs that lay dormant for years before being discovered by AI-assisted processes?

Re: “Next-token predictor” is the wrong mental model for LLMs

#140

Post by mofeien »

noduerme wrote: post #56
atleastoptimal wrote: post #31

Earlier quoted context omitted.

Yes this is correct. The thing is not about the term next-token predictor being correct, but because of the connotative weight of that phrase as a implicit trivialization of LLM abilities, which is how it is often used.

What is the motivation behind advocating against people trivializing LLMs? As in, why do you care?

Not the parent, but this incorrect trivialization of LLMs is often employed as a counterargument to the risks of AI such as "will take your job" or "will escape human control (again and worse)" or just "can possibly hurt me". And taking the easy feel-good cop-out instead of actively engaging with these questions is just.. harmful?
Post reply on HN