[flagged]
Does code cleanliness affect coding agents? A controlled minimal-pair study
91–100 of 112 posts
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#92Earlier quoted context omitted.
You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also…
You don't need the pre commit hooks if you can make the harness hooks.
Belt and suspenders. What if you make a manual code edit? What if your harness hook malfunctions?
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#93[dead]
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#94Did the degraded variants also end up with weaker test coverage, or was that controlled separately? Because the fix for "agent burns more tokens understanding the code" is very different from "agent gets confused by dead code" — the first needs better tests and docs, the second just needs a linter pass.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#95In the real world, clean code still has value.
In AI labs, they can run a task repeatedly for extended periods, eventually marking it as completed in benchmarks. But in the real world, completing tasks quickly and well remains important and valuable.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#96[flagged]
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#97Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…
If you're measuring a sloppy project against a well structured one with regard to how many tests fail, that'll bias success towards the sloppy codebase, which likely has worse test coverage and less robust testing in the first place. You'd essentially need to write a single test suite that works for both projects in each pair to compare fairly.
That's not to say the study is good, but I can respect their decision because the tests passing isn't necessarily correlated with the effectiveness of the agent.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#98Earlier quoted context omitted.
Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.
I have the agent inject comments that mention that this particular code is legacy and must not be used as a reference, should not be cleaned up, etc. If you have a document that lists all of the reasons not to use or touch some code, the comments can simply be references to it. // LEGACY CODE, per docs/legacy_rules.md §14, §19
Thanks for the tip. I’ll try that out.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#99[flagged]
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#100In similar work we're conducting at NJIT, we've found similar results. (We call it "Contextual Quality Contagion") Some of the most interesting comments here are about some of the real-world scenarios that are commonplace in industry: "mixed" quality codebases, as well as codebases with a mix of legacy code patterns with newer "better" patterns, causing the agents to confuse the conventions.
The minimal-pair design is honestly one of the strengths, as it tries to isolate the cleanliness from other factors (e.g. architecture, dependencies, tests) which is more rigorous than comparing repos. However, using LLM-generated "sloppified" code (rather than some kind of mechanical or human-guided) is a bit questionable.
I'd say the biggest critique which others have correctly mentioned here is the authors' choice to not check the full test suite. The claim of "behavioral equivalence" is only as good as the tests and coverage.
This hypothesis is compelling for two reasons: 1) it makes sense (garbage in -> garbage out as it were) since LLMs will mimic what they see in the codebase; 2) it matches what many engineers feel somewhat intuitively over the last year or two of using these models. Greenfield is almost always easier than joining a busy codebase. The mess comes in complex integration, and maintaining a system for legacy purposes, etc.