21–30 of 82 posts
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.
It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?
I know people, who initialize all required variables and write the logic after. which used to feel bonkers to me until I realized, they've done enough practice and memorization to be able to figure what they would need 10 steps down the line.
this does show that, models have a better model of the task and the expected end state.
The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty crude things like "is this program well-formed" and "is this program correct (will it pass tests)".
The paper says that these program "properties" (beliefs) predict future state of the program up to 25 "steps" ahead, but given the setup this seems to be expected. An agent is trying to fix a program and/or maintain it in a working state, so it doesn't seem surprising that current well-formedness and correctness persist into the future, or that the model is correctly "optimistic" about the outcome of the next action it is planning/predicting.
This incremental belief building from partial observability reminds me of the ability of LLMs to predict valid chess moves when only given a truncated history of the games moves so far (e.g. last 20 moves, not all moves back to start of the game).
It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?
...by inferring both the imports and the script body from the same context? I think you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports, but I don't see why that would be necessary. Infer imports from context, infer body from context + imports. All strictly causal.
> you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports
Yes, I am suggesting this. I don't think it is possible to write programs without either anticipating what you're going to write down below before you get there, or else being able to go back and edit what you already wrote.
Of course agent harnesses allow the latter, but raw models without a harness can still do an exceptionally, superhumanly, good job of straight-line programming with no editing.
> Infer imports from context, infer body from context + imports. All strictly causal.
Of course it's causal, that's kind of a reductive way to look at it.
Just infer the entire program from context and then type what you inferred.
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.
I’m still hesitant to interpret this as “thinking ahead” without at least seeing some more back-and-forth in the literature first, though. This just seems like one of those spots where it makes sense to give other researchers some time to come up with additional hypotheses to explain the observations instead of focusing on the first one anyone proposes in isolation.
It's been known for several years that LLM activations encode future tokens ahead of time (e.g. https://arxiv.org/abs/2404.00859 ). But this has only been shown on simple tasks, so I think this paper is still quite neat. The interesting thing is that they show "future horizon length" varies across models.
Of course, an interesting question what part of this internal computation is modeling for the future compared to guessing based on the given context (the past).
Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.
Earlier quoted context omitted.
> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.
At some level of performance reasoning becomes the most effective method to predict the next token