Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does code cleanliness affect coding agents? A controlled minimal-pair study

arxiv.org

Post reply on HN
Print view View original

11–20 of 112 posts

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#11

Post by softwaredoug »

rgoulter wrote: post #10

Another consideration: written by hand, the trade-off of development velocity vs well organised code means that it can be worth taking some tech debt now in order to deliver some value now. (Especially when prototyping etc.). With coding agents, agents can produce code quicker. The same trade-off still applies.. but, the time it takes an LLM coding agent to write well organised code is still going to be quicker than…

Yes this is one reason IMO I think of AI code as instant legacy code.

You take on a lot of tech debt. Then you need to do the same work you would do with any legacy app: finding where the brittle points are, what needs better testing, which leads to breaking apart the big ball of mud into cleaner components.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#12

Post by dnautics »

i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

i mean this is feeling too but im too paranoid and frequently do refactoring and code organization passes and never don't do it, so i cant say i know for sure there is a delta.

though people who complain that llms aren't that great strike me as the type to have messy code bases

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#13

Post by ngsevers »

softwaredoug wrote: post #7
ngsevers wrote: post #5

Even if agents can learn to navigate all the stubs and WET crap they leave behind do we really want a code base that no human can follow what's actually going on?

Even if the agent does everything, English is an imprecise description of what the code does. So I personally at a minimum will want to talk “in code” about what code does.

Sure, but that only happens with a clean, concise, human readable codebase. If agents start working directly in binary, it'd be a lot harder to interact and understand what's going on and where the points of failure are..

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#14

Post by BobbyTables2 »

i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

Feel the same way myself when working in messy codebases… At some point, the horrible patterns start to rub off…

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#16

Post by hannofcart »

i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems.

You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most mileage with agentic coding.

I wrote down a more detailed post of the various linters I use here:

https://www.balajeerc.info/Use-Deterministic-Guardrails-for-...

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#17

Post by smallerize »

ramraj07 wrote: post #9
wgd wrote: post #4

"agent pipelines that [...] clean a messy [repository]" This feels like a terrible approach, sufficient to condemn the entire study. Apparently half of the "minimal pairs" in this work were constructed in this way. I simply am not going to trust any conclusion that requires assuming these AI "cleaned" repos are in any way representative of actually-good codebases.

Would you trust clean repos that are messed up by AI?

No, because the outputs will still be "in distribution" so to speak.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#18

Post by yoyohello13 »

i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.
Post reply on HN