Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does code cleanliness affect coding agents? A controlled minimal-pair study

arxiv.org

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#81

Post by Muromec »

maccard wrote: post #56
jaggederest wrote: post #27

Earlier quoted context omitted.

It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.

> actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, You’ve never had an agent completely lose the plot and forget/confuse its instructions due to the context filling up?

That's the moment it should take a nap to compress the context. If you are expose the context size to the agent through some diagnostic input it will even do it itself

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#82

Post by Muromec »

NitpickLawyer wrote: post #40
BobbyTables2 wrote: post #14

Earlier quoted context omitted.

Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…

Every time this subject comes up, there are a bunch of takes along the lines of "would you work on a codebase maintained by agents? they'll mess up the code". And I'm asking myself where these people work, because in 20+ years I've yet to see that pristine state of a project that keeps being pristine after the honeymoon greenfield phase, and 50+ people start working on it. Every project devolves in time, old stuff ge…

Entropy is real and with offshore contractors you don't need agents to blame for it

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#84

Post by maccard »

DCKing wrote: post #68
maccard wrote: post #60

Earlier quoted context omitted.

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also…

> , as it will enforce those constraints no matter whether an agent or human is making the commit.

Unless the agent, or the human don't enable the precommit hooks in the first place.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#85

Post by marginalia_nu »

jkwang wrote: post #59

Interesting to see this quantified. Clean structure seems to lower the cognitive load for both humans and agents, which probably explains why naming and modularization matter more than we think.

The way we should define code quality is arguably how easy it is to affect correct changes to the code, that's hard to quantify, but ultimately the thing any code quality metric is trying to capture.

Based on that, it should be a pretty unsurprising conclusion as long as the code quality metrics you are using are reasonable; as long as the quality metric is good (within the context of coding agents), then this is the result we'd expect to see.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#86

Post by nicce »

gitaarik wrote: post #78
minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

I just say "refactor the codebase" and that also works pretty good! I have my code styling rules in my CLAUDE.md already anyway

> I have my code styling rules in my CLAUDE.md already anyway

Well... that is quite different than just saying "refactor the codebase".

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#87

Post by CuriousSkeptic »

geraltofrivia wrote: post #58
wgd wrote: post #4

"agent pipelines that [...] clean a messy [repository]" This feels like a terrible approach, sufficient to condemn the entire study. Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

First author here. Please let me offer a clarification. Our notion of "clean" isn't to just ask the agent to write better code. Rather, we give it a list of 50-100s static analyzer rule violations (and code LOC), and ask to remove them. We then check if the rule violations are resolved. Using LLMs to rewrite code to remove these violations is a rather accepted practice. Sonar's existing one-shot LLM based approach [1…

I do wonder though, how does metric based factoring compare to expert intuition based factoring? Can the latter be emulated by agents? Are there studies?

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#88

Post by softwaredoug »

minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

I can totally see doing this incrementally, but this seems extremely risky to do for the entire codebase in one shot on anything in production. Especially if you don’t have really thoughtful e2e tests of the whole system.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#90

Post by softwaredoug »

root_axis wrote: post #38

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…

At best the conclusions are a best case scenario for degraded code quality. That things remain functionally OK with more costly token usage to get work done.
Post reply on HN