Looking at the way they defined their test criteria, these seem like things that could be predicted with better than 50 percent accuracy by a trivial strategy. Am I mistaken? If I predict that on most edits, the program will not be fully correct but the number of passing tests will have gone up without regressing, then I ought to be correct a good deal of the time as long as the coding agent is half decent. I would expect that strategy to be less effective at predicting well-formedness, but that is also by far the worst-performing criterion they evaluated.
Coding agents think ahead of time
51–60 of 82 posts
Re: Coding agents think ahead of time
#52Earlier quoted context omitted.
I keep asking the same question, and I think the steelman version would be "has metacognitive patterns similar to humans"
I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all. This just changes the argument from LLMs can reason like a human to LLMs can't reason at all while ignoring the third possibility of "LLMs can reason not like a human". One of the biggest things I've learned after the event of LLMs is that humans definit…
> I think the anthropomorphic view of this is dangerous in the long term as it starts the argument that anything that isn't reasoned by a human isn't reasoning at all.
Something like this is actually a stance in the tradition of inferentialism (see the term sapience). Though "reasoning" isn't like, turing machine computability in this space; from what I understand, it's some abstract notion of the "space of reasons". I don't really understand it, honestly.
There's some merit to this, IMO. When an LLM goes wrong, do you blame the person or the LLM? As in, would you throw said LLM in jail, and hold the LLM accountable? Not right now, at least. I'm not sure if that's what is meant by the "space of reasons", but the intuition is that 'reason' can mean a lot of different things, pragmatically speaking. Reason as a legible audit trail is one of those ways.
But that's arguably getting into the social aspect of 'reason' (important!) and not like, what STEM people traditionally think of as 'reason'.
Re: Coding agents think ahead of time
#53Earlier quoted context omitted.
I tend to agree but to adjudicate that someone has to define what they consider reasoning to be.
Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions. The other approach, very popular on HN, has led to endless carping by people making outlandish claims without evidence and then demanding that every term possible be defined before they can be dismissed out of hand. The claim that any two things are equivalent just because they look similar is a strong one, and…
> Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions
Ok, I claim that if something draws reasonable conclusions to questions it hasn't previously seen by performing steps that look like reasoning, then it is reasoning.
If you disagree then please define reasoning.
Re: Coding agents think ahead of time
#54This feels to me more like incremental belief building than "thinking ahead of time" (which is not what the paper is claiming). The model only has partial observability of the program it is working on (whatever tool call outputs are present in the context), as well as the trajectory of actions it has taken, and from this is building up some internal beliefs about the program - the probes used were looking for pretty…
> more like incremental belief building than "thinking ahead of time"
concretely, what's the difference here?
(I suppose you could define "thinking ahead of time" as explicitly using something like "thinking tokens" which might be roughly analogous to system1/system2 thinking, but note that we still call system 1 thinking "thinking")
Re: Coding agents think ahead of time
#55It makes intuitive sense. How else could you write a 500-line script top-to-bottom with no backspace key and no arrow keys and get all the imports etc. right upfront?
...by inferring both the imports and the script body from the same context? I think you're suggesting there's some kind of information flow from the anticipated body of the script back up to the imports, but I don't see why that would be necessary. Infer imports from context, infer body from context + imports. All strictly causal.
That suggests a straightforward mapping between context -> program. Maybe for trivial programs but I don’t see how you can argue that for sufficiently complex programs.
Reasoning models perform better than non-reasoning models because they’re able to refine their code in multiple steps. That allows any part of the program to influence any other part of the program, not just from start -> end.
Human thinking serves a similar purpose. Basically intelligence needs to be able to backtrack if you want better performance.
Re: Coding agents think ahead of time
#56Earlier quoted context omitted.
Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions. The other approach, very popular on HN, has led to endless carping by people making outlandish claims without evidence and then demanding that every term possible be defined before they can be dismissed out of hand. The claim that any two things are equivalent just because they look similar is a strong one, and…
> Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions Ok, I claim that if something draws reasonable conclusions to questions it hasn't previously seen by performing steps that look like reasoning, then it is reasoning. If you disagree then please define reasoning.
This is actually a perfect example of what I was referring to in my earlier comment; the strident [1] and rather aggressive insistence that your unsupported belief be proved wrong, rather than you providing literally any evidence that two distinct things are identical other than the assertion itself.
You're making a strong claim here; the burden of proof [2] lies with you, in much the same way that it would if you declared that horoscopes know the future. We have a sufficient explanation for LLM behaviour that we have no reason to discard based solely on your whim; if you want to be convincing, then you'll need more than assertion.
Similar appearance does not mean identical nature, and your position does not warrant serious consideration until you provide support for it.
[1] https://en.wikipedia.org/wiki/Sealioning [2] https://en.wikipedia.org/wiki/Burden_of_proof_(philosophy)
Re: Coding agents think ahead of time
#57Earlier quoted context omitted.
> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.
Save us from the reasoning / sentience / consciousness / thinking semantic quicksand. Babbage’s Analytical Engine didn’t actually analyze anything, and terminology hadn’t gotten any more clear-cut since.
+1. Inputs, processing & memory do the job.
I suspect exact and/or universal definitions for intelligence, self-awareness, 'feelings' etc will prove to be elusive, and best we'll get is systems/robots etc behaving as if possessing those qualities. With some tests to put a number on them.
Downside is that may apply to us humans too.
Re: Coding agents think ahead of time
#58Earlier quoted context omitted.
> Crucially it's the responsibility of those who are making the claim here to bring evidence and definitions Ok, I claim that if something draws reasonable conclusions to questions it hasn't previously seen by performing steps that look like reasoning, then it is reasoning. If you disagree then please define reasoning.
This is actually a perfect example of what I was referring to in my earlier comment; the strident [1] and rather aggressive insistence that your unsupported belief be proved wrong, rather than you providing literally any evidence that two distinct things are identical other than the assertion itself. You're making a strong claim here; the burden of proof [2] lies with you, in much the same way that it would if you de…
I provided my definition and argument. I can't engage further if you don't refute any of it.
Re: Coding agents think ahead of time
#59Earlier quoted context omitted.
> To effectively predict the next token it needs a good idea of what comes after the next token. And that's all it needs. Not reasoning.
What’s the difference?
Is regexp or compiler lookahead reasoning? No.
Mistaking chatbot lookahead as reasoning comes with being gulled by the "artificial intelligence" sales pitch.