Coding agents think ahead of time
arxiv.org
Coding agents think ahead of time
1–10 of 82 posts
Re: Coding agents think ahead of time
#2Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
Re: Coding agents think ahead of time
#3Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
Isn't that what "Attention is all you need" was about anyway? Does not sound like news to me.
Re: Coding agents think ahead of time
#4How is this news? isn't it an obvious fact from the Transformer architecture?
Re: Coding agents think ahead of time
#5> A coding agent solving a software-engineering task spends dozens of steps reasoning
No. That's simple PR hype. Parrotry is not reasoning.
Re: Coding agents think ahead of time
#6Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
> To effectively predict the next token it needs a good idea of what comes after the next token.
And that's all it needs. Not reasoning.
Re: Coding agents think ahead of time
#7How is this news? isn't it an obvious fact from the Transformer architecture?
[deleted]
Re: Coding agents think ahead of time
#8>> Probes trained to predict the outcome of future edits (before they are materialized and written on disk) achieve performance above chance up to roughly 25 steps in advance.
Are these probes effectively run in parallel? The way this reads is more about predicting a future outcome than keeping the current token relevant based on past tokens.
Re: Coding agents think ahead of time
#9Confirmatory of Sutskever's view that predicting the next token forces a deep understanding. To effectively predict the next token it needs a good idea of what comes after the next token.
I think that's reading a little too much into it. The paper shows the hidden states contain signals about whether the code is good right now, and whether the run is probably going to work out. That's interesting, for sure. But it doesn't mean the model has some detailed idea of what it's going to write 25 steps later.
A lot of that signal could be much simpler stuff. This task is hard. The agent seems stuck. The tests are getting better. The current approach looks promising. All of those things make future success easier to predict without the model actually "knowing what comes next" in any strong sense.
Also, their 25 steps are agent turns, not 25 code edits. The median run had something like 52 steps but only two edits, and the program label stays the same between edits. So "25 steps ahead" may sometimes just mean basically the same codebase, with a bunch of reading and test output in between.
So yeah, I'd say it's consistent with Sutskever's view. But "consistent with" and "confirmatory of" are doing very different amounts of work here.
Re: Coding agents think ahead of time
#10How is this news? isn't it an obvious fact from the Transformer architecture?
[deleted]