Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does code cleanliness affect coding agents? A controlled minimal-pair study

arxiv.org

Post reply on HN
Print view View original

41–50 of 112 posts

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#41

Post by minimaxir »

loremium wrote: post #29
minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

the word your looking for is idiomatic

Pythonic is a specific type of idiomatic; it never hurts to be overly specific with modern agents.

Also, as evident from the different tones in these two prompts, my prompts are often freeform.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#42

Post by vetronauta »

ramraj07 wrote: post #8
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

I was reading your comment, agreeing with it but still feeling why this is a bad comment. It just occurred to me that an anecdotal statement like this is the antithesis of scientific discourse. We have a paper here, trying to answer a question, and anecdotal testimonials can only harm the discussion by biasing readers without adding anything of value to let anyone objectively conclude anything on the problem. The mos…

I was reading your comment, disagreeing with it but still feeling why this is a good comment. It just occurred to me that this is not science: science must be reproducible and this is just an historical report on artifact that will be unavailable soon.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#43

Post by maxglute »

As pure slop coder, after telling agent to clean up my messy projects from earlier models, i.e. ts, 400 line max, and most of my tasks have gone down to consuming fraction of quota as previous and tasks takes significantly less time to finish. I also just tell it to evaluate tasks on compute/token efficiency, i.e. whether to reuse context, or smart new convo/switch models, and now my quota stretch much further. Like I assume agents should be smart enough to route tasks to proper model capabilities in background, but then I ask myself... what economic incentive to they have but to make users token max.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#45

Post by minimaxir »

hatefulheart wrote: post #35
minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

Have you tried telling it: “Write perfect code, make no mistakes” I use this one in my Ralph Harness all the time, it’s a classic! It’s not that it can’t do that, it’s just that you haven’t told it to!

That's not what this is. Refactors are necessary because LLMs tend to bloat even with controls against it, but it's ok to make mistakes if they can be easily fixed.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#46

Post by hatefulheart »

minimaxir wrote: post #45
hatefulheart wrote: post #35

Earlier quoted context omitted.

Have you tried telling it: “Write perfect code, make no mistakes” I use this one in my Ralph Harness all the time, it’s a classic! It’s not that it can’t do that, it’s just that you haven’t told it to!

That's not what this is. Refactors are necessary because LLMs tend to bloat even with controls against it, but it's ok to make mistakes if they can be easily fixed.

So wait you’ve told it not to make any mistakes or you haven’t? Seems like you’re missing this one easy trick!

It’s not about making mistakes, it’s about telling it not to make mistakes!

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#47

Post by saghm »

hannofcart wrote: post #16
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems. You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most…

Sure, giving agents a deterministic way to run linters is good, but that already implicitly accepts the premise that code hygiene does actually matter. If you agree with that, then you probably also recognize that there are some things that you can't lint for because they're a lot harder to define, and why would you expect an agent to be able to figure that out today when they still need our intervention to even properly run linters?

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#48

Post by localhoster »

If we don't write the codebase, we don't write the tests, and the agents are modifying it, what even considered a messy codebase? Too many files? Too little files? Related code spread accros many files? What is considered a messy codebase?

Have you even seen one ever?

Also, not checking if it breaks unrelated tests is wild, good software is written by modifying the "just right" amount of code to get your result.

If you break _unrelated_ tests than you most likely didn't changed the right amount of code.

Idk I call bs.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#49

Post by jwpapi »

root_axis wrote: post #38

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…

Also controlling input

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#50

Post by yoyohello13 »

rafaelmn wrote: post #25
hannofcart wrote: post #16

Earlier quoted context omitted.

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems. You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most…

> Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems. I have legacy endpoints that are no longer used in practice, there for historical reasons, intertwined with existing code etc. They might be marked obsolete, services implementing it are not - agent greps those, builds off of…

Yeah, we have a big struggle with this. We have lots of legacy code that doesn't follow our latest design patterns intermixed with new code. The LLM picks up bad habits depending on what it pulls in to context first. We have AGENTS.md configured with the right way, but old style still slips in. We obviously need to update the old code but on the other hand if it ain't broke why touch it.
Post reply on HN