Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Does code cleanliness affect coding agents? A controlled minimal-pair study

arxiv.org

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#71

Post by supermatt »

A lot of those tokens are traversal - either searching for code or following call-sites. Basically building enough context to be able to work on the task.

You can reduce a lot of the token use for traversal by giving your agent access to some form of LSP in addition to hierarchical direction with your AGENTS.md (or equivalent) for monorepos - but a spread-out codebase is always going to end up requiring some form of traversal to solve each task.

And that traversal isn't just token use - its repeated round trip latency (LLM (queue time -> prefill -> decode -> output) -> Agent (parsing -> tool call -> tool response) -> back to LLM) for EACH step (well, some can be done in parallel, but in practice its mostly sequential) - slowing down the task considerably.

Locality and structure are key when it comes to efficient use of agents. The context window is always bounded and attention across it is inconsistent.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#73

Post by rullelito »

jwpapi wrote: post #49
root_axis wrote: post #38

Interesting question to study, but I'm extremely skeptical of the experimental design. They used Opus 4.6 to synthetically produce "degraded" or "cleaned" code bases for relative comparison in the experiment. Worse, they don't control for breaking the application's tests. > Pass rate scores the agent’s final state against the hidden tests we wrote for each task. We do not check whether the agent broke unrelated tests…

Also controlling input

Input matching distribution of a specific one-shotted LLM... Nice.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#74

Post by embedding-shape »

yoyohello13 wrote: post #18
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

I’ve been working with these things for quite some time now and every time I simply “treat it like I would a human” it seems to perform better. I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one. Just like it performs better when it has well formed specs and access to documentation.

> I can’t imagine agents wouldn’t perform better in a clean codebase than a giant mess of one.

I guess it depends on what you mean with "better" but almost all the agent-built projects I do with zero regards to code quality, design and architecture ends up with every single agent needing 10+ minutes to do even the easy changes, while the ones where I focused on those things together with the agent, large changes can take 10+ minutes but everything else is solved faster.

I don't have empirical evidence of this yet, I guess I should put together some sort of test to confirm/disconfirm this.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#75

Post by darkerside »

Tarq0n wrote: post #67
dprkh wrote: post #51

Earlier quoted context omitted.

I think in this context it is actually important to share exact wording that causes the AI to perform well. My favorite is "Do not use your own knowledge."

That's positively incoherent. You can't even interpret an instruction without relying on prior knowledge. Not to mention LLM's are in essence made of knowledge.

No, it's a specific direction to look information up in docs or online rather than relying on model training.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#76

Post by gitaarik »

hannofcart wrote: post #16
i_have_an_idea wrote: post #3

In my experience, the delta in agent performance is substantial if the codebase is littered with dead code, redundant code, unreachable fallbacks, leaking abstractions and half-baked design patterns vs if the code is well-organized, with clear data flow, with good encapsulation and clean architecture. Like, I've seen all the frontier models have to do several rounds of code review / QA and fix when the code is bad vs…

Some of the issues mentioned above like dead code removal, code duplication, unreachable code are already solved using deterministic linters for quite a while now for most language ecosystems. You can get the LLM to run a script which checks for all of these and also enforce them by running the same script as a pre-commit hook. Setting this up religiously in every code base I work on has been what's given me the most…

Why not just fix the code before you make a new feature? Or is that what you mean? Only you automatically fix the code on every prompt?

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#77

Post by jillesvangurp »

minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

Exactly. Simply asking agentic coding tools to clean up code bases, to do some targeted refactorings, to enforce things like SOLID principles, and other good practices can result in a lot of easy improvements.

I've noticed a thing where by default, agentic coding tools are reluctant to remove code. Even when you tell them to. It will bend over backwards to keep old code around, to add complexity for allowing that code to still be called, etc. Super annoying if you are basically just prototyping. You basically end up with a lot of dead code, which than confuses things when you try to add to it. But once you know this, you can just ask it to get rid of the legacy stuff.

Keeping the code base clean, actually stimulates AIs to do the right thing. If there are lots of tests, it will add more when creating new functionality. If there's documentation, it will update that without needing to be prompted as well.

As code harnesses improve, a lot of this is probably being built in as well. Which means even less experienced prompters can get decent results.

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#78

Post by gitaarik »

minimaxir wrote: post #21

One trick I've found that works well is to tell it to refactor, e.g for Python: Refactor the Python code to make it more Pythonic, e.g. fewer classes/singletons, especially if it will provide a speedup. The Python code **MUST** follow code organization standards expected of popular open-source Python packages code without causing any benchmark performance regressions. A variant I've used for Rust code: The Rust codeb…

I just say "refactor the codebase" and that also works pretty good!

I have my code styling rules in my CLAUDE.md already anyway

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#79

Post by Muromec »

maccard wrote: post #60
Yokolos wrote: post #57

Earlier quoted context omitted.

This is the most frustrating part. You do everything you can to ensure there are clear instructions, you can keep the agent MD as concise and clear and short as possible. It still feels like it's all just a suggestion, and of course it is, because it's all just another part of the prompt.

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

Writing a custom harness was like 10 bucks of the communist model tokens last month and I ended up with an a actor library, a thing that has a stable identity, writes sutras and knows it wrote the harness itself. Weird.

Adding an auto runner of the unit tests is just... Boring?

Re: Does code cleanliness affect coding agents? A controlled minimal-pair study

#80

Post by Muromec »

DCKing wrote: post #68
maccard wrote: post #60

Earlier quoted context omitted.

I’m in the “AI can be great, but it’s not right now” camp. I think that pulling the verification into the harness and having the harness execute it rather than the agent would genuinely make AI go to usable for me. But even prototyping a custom harness requires API billing which is just so expensive…

You can put many agent constraints in precommit hooks if they're static checks. I ask agents to make commits, and e.g. in a Python project have the precommit hook fire off type checks, linting and even architectural things like import boundaries (using `tach`). When an agent is prepped to make commits themselves, it will catch pre-commit failing and correct itself. The existence of static checks themselves might also…

You don't need the pre commit hooks if you can make the harness hooks.
Post reply on HN