Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coding agents think ahead of time

arxiv.org

Post reply on HN
Print view View original

71–80 of 82 posts

Re: Coding agents think ahead of time

#71

Post by HarHarVeryFunny »

dnautics wrote: post #54
HarHarVeryFunny wrote: post #24

This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming). The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty…

> more like incremental belief building than "thinking ahead of time" concretely, what's the difference here? (I suppose you could define "thinking ahead of time" as explicitly using something like "thinking tokens" which might be roughly analogous to system1/system2 thinking, but note that we still call system 1 thinking "thinking")

Well to be fair to the author "thinking ahead of time" wasn't how they described it - that is just the HN title. "Thinking ahead" sounds like the model reasoning about the future state of the program, which is fundamentally unknown. The most the model could have learnt during RL-training, if trained in a harness, is what kind of outcomes to generally expect such as agentic progamming generally being successful (trend towards well-formed and passes tests), except perhaps for certain context smells where it appears to be going down the tubes!

My hunch is that the model isn't thinking/planning head in order to achieve these probe results, but rather that they just reflect the models current take/beliefs on "how it's going" right now, and any correlation with "how it's going" in 25 steps time is just due to the inherent biases in the setup - well-formed programs tend to stay well-formed, etc.

Re: Coding agents think ahead of time

#72

Post by chrisjj »

HarHarVeryFunny wrote: post #67
chrisjj wrote: post #65

Earlier quoted context omitted.

> Copying someone else's reasoning process (as best you understand it) is still a limited form of reasoning The LLM is not copying someone else's process. All it knows it someone else's product , and its only process is to compute the most likely next token within that.

True, and this is why I said "as best you understand it". The visible (copyable) circumstances in which someone makes a reasoning decision aren't the same as the internal decision making process, so this is just cargo-cult copying - copy the surface form without understanding the reason for it. The perhaps surprising thing is that it works so well some of the time - specifically for math and coding.

Its surprises people just as did a web seach 30 years ago. "Wow, it found my page amongst millions - in seconds!" Just because raw dumb huge computer power is beyond the average user's compehension.

Re: Coding agents think ahead of time

#73

Post by coldtea »

chrisjj wrote: post #45
coldtea wrote: post #39

Earlier quoted context omitted.

You keep repeating this throughout the thread, one could even say just like a certain class of exotic birds :) Why couldn't parroting, after a certain level and complexity of the parroting infrastructure, be reasoning? What a priori restriction forbids it from being such? Flipping a NAND is not calculation either. Billions of them? Things change.

> Flipping a NAND is not calculation either. But it is.

Well, processing a weighted model to respond is reasoning too.

Re: Coding agents think ahead of time

#75

Post by chrisjj »

wren6991 wrote: post #69
CooCooCaCha wrote: post #55

Earlier quoted context omitted.

That suggests a straightforward mapping between context -> program. Maybe for trivial programs but I don’t see how you can argue that for sufficiently complex programs. Reasoning models perform better than non-reasoning models because they’re able to refine their code in multiple steps. That allows any part of the program to influence any other part of the program, not just from start -> end. Human thinking serves a…

Right, reasoning models are almost more notable for what they don't do, which is: try to generate the first token of the response from a single forward pass through the weights. Non-reasoning models were still surprisingly good at generating working programs. I think it makes sense when you consider that it's still iterative; it doesn't have to generate the entire import block in a single pass, for example, more like…

> you see LLMs use a lot of scoped imports and I think this is partly because it makes the output more resilient against failure to think forward at the start.

What makes you think an LLM has any such motivation? Or any motivation at all - beyond generating the most likely next token?

Re: Coding agents think ahead of time

#76

Post by wren6991 »

chrisjj wrote: post #75
wren6991 wrote: post #69

Earlier quoted context omitted.

Right, reasoning models are almost more notable for what they don't do, which is: try to generate the first token of the response from a single forward pass through the weights. Non-reasoning models were still surprisingly good at generating working programs. I think it makes sense when you consider that it's still iterative; it doesn't have to generate the entire import block in a single pass, for example, more like…

> you see LLMs use a lot of scoped imports and I think this is partly because it makes the output more resilient against failure to think forward at the start. What makes you think an LLM has any such motivation? Or any motivation at all - beyond generating the most likely next token?

The fact that RLVR rewards successful trajectories.

Notice I didn't use the word motivation, which you decided to use. The reason LLMs use a technique is simply because it works, and that comes straight from training (specifically RLVR post-training).

Re: Coding agents think ahead of time

#77

Post by buzzin__ »

Planktonne wrote: post #56
rlt wrote: post #53

Earlier quoted context omitted.

> Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions Ok, I claim that if something draws reasonable conclusions to questions it hasn't previously seen by performing steps that look like reasoning, then it is reasoning. If you disagree then please define reasoning.

This is actually a perfect example of what I was referring to in my earlier comment; the strident [1] and rather aggressive insistence that your unsupported belief be proved wrong, rather than you providing literally any evidence that two distinct things are identical other than the assertion itself. You're making a strong claim here; the burden of proof [2] lies with you, in much the same way that it would if you de…

It's simply amazing how you can be so confidently wrong.

You got a perfectly reasonable response, clear, short. Yet, you chose to ignore it and attack the guy by claiming that the fact that LLMs give a reasonable response are his "beliefs", and "unsupported" at that. LoL.

Is you claim that LLMs don't give a reasonable answers? Did you try to ask them some complex questions in the last 12 months?

Or is your line of reasoning more in the line of "they heard all the questions before, and just memorized the answers?"

Or, do you not agree with a statement that giving reasonable answers is reasoning?

Re: Coding agents think ahead of time

#78

Post by vidarh »

anuramat wrote: post #31
vidarh wrote: post #16

Earlier quoted context omitted.

How do you define reasoning in a measurable way?

I keep asking the same question, and I think the steelman version would be "has metacognitive patterns similar to humans"

I think you're overestimating the thinking here. I ask this question very often, and the persons I ask if of practically never answer.

I ask it to highlight that most people have no clear answer to what we mean by reasoning. It's a notoriously hard term to define with an precision, and even harder to define by precision in a way that would exclude LLM's, which is usually what the people I ask it of want to do.

I don't think your steelman version really "works" for that purpose because the immediate response would be that it makes "reasoning" largely irrelevant if you define the term that restrictively.

Re: Coding agents think ahead of time

#80

Post by Planktonne »

buzzin__ wrote: post #77
Planktonne wrote: post #56

Earlier quoted context omitted.

This is actually a perfect example of what I was referring to in my earlier comment; the strident [1] and rather aggressive insistence that your unsupported belief be proved wrong, rather than you providing literally any evidence that two distinct things are identical other than the assertion itself. You're making a strong claim here; the burden of proof [2] lies with you, in much the same way that it would if you de…

It's simply amazing how you can be so confidently wrong. You got a perfectly reasonable response, clear, short. Yet, you chose to ignore it and attack the guy by claiming that the fact that LLMs give a reasonable response are his "beliefs", and "unsupported" at that. LoL. Is you claim that LLMs don't give a reasonable answers? Did you try to ask them some complex questions in the last 12 months? Or is your line of re…

I think you're struggling her with the distinction between a statement and evidence for the statement. I know what the commenter believes, but they don't have any evidence for it beyond appearance.

The appearance of similarity in two things we can already explain without having to posit identical nature is not proof that they are identical. Screwdrivers and hammers, planes and birds: we know about all sorts of things with similar effects but different mechanisms.

If you want to argue for more similarity, you need more evidence than just your belief and superficial resemblance. That's got nothing to do with the results or capabilities of LLMs (and it's either blinkered or disingenuous that you're trying to elide that).

> do you not agree with a statement that giving reasonable answers is reasoning?

Note the seventh word: it's a statement. The other commenters knows it's a statement, you know it's a statement, not an argument or an explanation. but you are seemingly desperate for people to accept it as a proven fact without putting in the effort to prove it.

It's unscientific and frankly embarrassing to be so certain without being able to do anything but hector about it; if it's so obvious, prove it.

Post reply on HN