[flagged]
Does code cleanliness affect coding agents? A controlled minimal-pair study
61–70 of 112 posts
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#62Earlier quoted context omitted.
Would you trust clean repos that are messed up by AI?
Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs. I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a su…
Our initial suggestion was to do something along the lines of your proposal. But we found that when we ask an agent to implement features for `t-n`^th PR, even when we are overly specific makes the code rather divergent, to the point that sometimes the `t-n+1`th task description doesn't really make a lot of sense.
In fact, there are some papers (that we cited) which create a set of tasks doing exactly this, and it is non-trivial [1].
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#63Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…
> they don't control for breaking the application's tests.
Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#64Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…
> they don't control for breaking the application's tests. Essentially it's an absolutely worthless study then. This AI fatigue got boring a long time ago, this is just painful now.
To be sure, we had hidden tests that validate whether agent implemented the task appropriately. And this is something we paid a lot of attention to (pass rate, in our paper.)
What we didn't do (stupid oversight on my part) was to ensure that there are no regressions in the remaining tests in the repo (unrelated to the current task). In practice, when using Sonnet 4.6 IRL, I don't see a lot of regressions because often the agent runs the test before calling it done. But it could have gone either way. We don't know.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#65I wrote this, where possible I use non llm tooling to assess and llm just writes up the findings.
https://github.com/bjcoombs/ai-native-toolkit/tree/main#what...
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#66[flagged]
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#67Earlier quoted context omitted.
The word you are looking for is "you're". (Can we not play language police? It's boring and doesn't lead to interesting discussion.)
I think in this context it is actually important to share exact wording that causes the AI to perform well. My favorite is "Do not use your own knowledge."
That's positively incoherent. You can't even interpret an instruction without relying on prior knowledge. Not to mention LLM's are in essence made of knowledge.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#68Earlier quoted context omitted.
This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.
I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…
You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also help agents gain awareness of the overall verification flow including larger things like tests, but that's hard to say for certain.
Putting structural code checks in a precommit hook is arguably better than pulling it into the harness, as it will enforce those constraints no matter whether an agent or human is making the commit.
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#69Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…
Also controlling input
[dead]
Re: Does code cleanliness affect coding agents? A controlled minimal-pair study
#70If we don't write the codebase, we don't write the tests, and the agents are modifying it, what even considered a messy codebase? Too many files? Too little files? Related code spread accros many files? What is considered a messy codebase? Have you even seen one ever? Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result. If y…
Paraphrasing: if I don't write the codebase [but someone else does], I don't write the tests, and other team members are modifying it, what even is considered a messy codebase?
I actually don't see a connection between the mechanism used to create the code and the code messiness. Things like code repetition, incorrect level of abstractions, tests testing only tests themselves, using too smart optimizations for things that don't matter, .. These all can happen in both person- and machine -authored code.
I would be surprised if a professional software developer has never seen at least some aspects of messy codebase in most any large project. The difference can be whether this messiness ever managed, or just piled on.