Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does code cleanliness affect coding agents? A controlled minimal-pair study

arxiv.org

Post reply on HN
Print view View original

21–30 of 112 posts

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#21

Post by minimaxir »

One trick I've found that works well is to tell it to refactor, e.g for Python:

    Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions.
A variant I've used for Rust code:

    The Rust codebase in `/src` has become bloated with several files >1k LoC. Refactor the Rust codebase to fit code organization standards expected of popular open-source Rust code without causing any benchmark performance regressions.
Those types of prompts appear to a) reorganize the code logically and b) do seem to get better performance from the agents because the file names now provide semantic hints to where relevant code resides. For bloated 5k LoC files, the agent has to Read several chunks to find relevant code which is inefficient.

In terms of benchmark performance it generally improves after the refactor which I suspect is coincidental (especially in Rust where it shouldn't matter due to compiling) but I'm not complaining.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#22

Post by Supermancho »

Gigachad wrote: post #6

I can't imagine how it wouldn't. None of them can fit a real codebase in context and have to browse the code the same way a person would. Doing searches and reading files. If the files are in the places they would be expected to be and things are called what the model or a person would first guess to search then it gets found in the first attempt rather than requiring a deep search and multiple attempts.

An LLM doesnt have to hold the whole codebase in context. Every path, then shift to next paths because you can ignore sections covered. Much like a developer would.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#23

Post by cpill »

minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

yeah, me too. I usually ask it to do a code review using SOLID standards and it usually does a good job, if not a little overkill sometimes.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#24

Post by wgd »

ramraj07 wrote: post #9
wgd wrote: post #4

"agent pipelines that [...] clean a messy [repository]" This feels like a terrible approach, sufficient to condemn the entire study. Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

Would you trust clean repos that are messed up by AI?

Yes, those ones would be at least a somewhat-plausible simulation of a real scenario people care about: a once-clean codebase that was allowed to become messy by a succession of insufficiently-careful vibeslop PRs.

I'm not a huge fan of their methodology for the AI-degraded cases either (ideally one would set up the mirror pairs by taking some real repositories and rewinding history a month or so and then having a succession of independent agents reimplement each bit of feature work and bugfixes over that period of time), but it's at least a coarse approximation whereas I just don't trust the cleanup methodology to resemble anything real in the first place.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#25

Post by rafaelmn »

hannofcart wrote: post #16
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems. You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most…

> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of them - produces half legacy garbage.

Linters only handle trivial cases most of us already solved.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#27

Post by jaggederest »

yoyohello13 wrote: post #18
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.

It actually goes even further than humans, humans can pretty rapidly get inured to things being awkward or messy and stop noticing, but the context for agents is taking up the same space and "attention" every time they're run, and they're creations entirely of context, so the quality and examples matter massively.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#28

Post by ford »

In my experience everything that affects engineers affects an agent. Good abstractions, reasonably sized methods, good names, principled (intra & inter) service architecture, unit tests, etc.

All of these things have historically been the job of engineers, because it helps other people contribute to the code.

Now it helps other people and other agents contribute to the code.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#29

Post by loremium »

minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

the word your looking for is idiomatic

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#30

Post by bckr »

loremium wrote: post #29
minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

the word your looking for is idiomatic

Yes and Pythonic is the Pythonic word for idiomatic
Post reply on HN