Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Over-editing refers to a model modifying code beyond what is necessary

nrehiew.github.io

Post reply on HN
Print view View original

131–140 of 267 posts

Re: Over-editing refers to a model modifying code beyond what is necessary

#131

Post by grttww »

slopinthebag wrote: post #111
grttww wrote: post #22

Earlier quoted context omitted.

When you steer a car, there isn’t this degree of probability about the output. How do you emulate that with llm’s? I suppose the objective is to get variance down to the point it’s barely noticeable. But not sure it’ll get to that place based on accumulating more data and re-training models.

Well, the point is by steering it you can get both more expected/reproducible output, and you can correct bad assumptions before they become solidified in your codebase. You can get pretty close to reproducible output by narrowing the scope and using certain prompts/harnesses. As in, you get roughly the same output each time with identical prompts, assuming you're using a model which doesn't change every few hours to…

You wrote all that and didn’t address the question lmao.

There’s diminishing returns and moreover this idea that people are holding it wrong / they need to figure out the complexity goes against all that has been done over the past 30 years : making things simpler.

Re: Over-editing refers to a model modifying code beyond what is necessary

#132

Post by ok_dad »

sdevonoes wrote: post #117
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

For that kind of flow, I prefer to work without AI.

The agent mostly helps me reduce cognitive load and avoid the fiddly bits. I still review and understand all of the code but I don’t have to think about writing all of it. I also still hand write tons of code when I want to be very specific about behavior.

Re: Over-editing refers to a model modifying code beyond what is necessary

#133

Post by ok_dad »

applfanboysbgon wrote: post #62
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

This is significantly slower than just writing the code yourself.

I don’t find it slower overall, personally, but YMMV depending on how you like to tackle problems. Also the problem space and the project details can dictate that these tools aren’t helpful. Luckily the code I write tends to be perfect for a coding agent to clank away for me.

Re: Over-editing refers to a model modifying code beyond what is necessary

#134

Post by raincole »

xboxnolifes wrote: post #127
raincole wrote: post #35

Earlier quoted context omitted.

Really? I've never heard it's considered wise to put refactoring and new features (or bugfixes) in the same commit. Everyone I know from every place I've seen consider it bad. From harmful to a straight rejection in code review. "Refactor-as-you-go" means to refactor right after you add features / fix bugs, not like what the agent does in this article.

Notice how they didn't say to put it in the same commit. The real issue, and why refactor as you go isn't done as much, is the overhead of splitting changes that touch the same code into different commits without disrupting your workflow. It's not as easy as it should be to support this strategy. Instead you to do it later, and then never do it.

I think you're talking about a different topic unrelated to the linked article. In the linked article the LLM doesn't split it into several commits. If LLM had a button to split the bug fix and the overall refactoring, the author wouldn't complain and we wouldn't see this article.

Re: Over-editing refers to a model modifying code beyond what is necessary

#135

Post by slopinthebag »

grttww wrote: post #131
slopinthebag wrote: post #111

Earlier quoted context omitted.

Well, the point is by steering it you can get both more expected/reproducible output, and you can correct bad assumptions before they become solidified in your codebase. You can get pretty close to reproducible output by narrowing the scope and using certain prompts/harnesses. As in, you get roughly the same output each time with identical prompts, assuming you're using a model which doesn't change every few hours to…

You wrote all that and didn’t address the question lmao. There’s diminishing returns and moreover this idea that people are holding it wrong / they need to figure out the complexity goes against all that has been done over the past 30 years : making things simpler.

You asked me how one could minimise the non-deterministic output of LLM's and I responded, if that's not good enough of an answer feel free to ask a follow up.

Re: Over-editing refers to a model modifying code beyond what is necessary

#136

Post by ok_dad »

wahnfrieden wrote: post #55
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

It’s terribly slow

I get it, but if tomorrow every inference provider doubled costs I still understand my applications code and can continue to work on it myself.

Re: Over-editing refers to a model modifying code beyond what is necessary

#137

Post by sornaensis »

ok_dad wrote: post #128
sornaensis wrote: post #125

Earlier quoted context omitted.

I define tools that perform individual tasks, like build the application, run the tests, access project management tools with task context, web search, edit files in the workspace, read only vs write access source control, etc. The agent only has access to exactly what it needs, be it an implementation agent, analysis agent, or review agent. Makes it very easy to stay in command without having to sit and approve tons…

This is a cool idea, can you write more about how your tools work or maybe short descriptions of a few of them? I’m interested in more rails for my bots.

I just made MCP servers that wrap the tools I need the agents to use, and give no-ask permissions to the specific tools the agents need in the agent definition.

Both OpenCode and VsCode support this. I think in ClaudeCode you can do it with skills now.

The other benefit is the MCP tool can mediate e.g. noisy build tool output, and reduce token usage by only showing errors or test failures, nothing else, or simply an ok response with the build run or test count.

So far, I have not needed to give them access to more than build tools, git, and a project/knowledge system (e.g. Obsidian) for the work I have them doing. Well and file read/write and web search.

Re: Over-editing refers to a model modifying code beyond what is necessary

#138

Post by sarchertech »

andoando wrote: post #121
pron wrote: post #112

Earlier quoted context omitted.

POC, sure (although 10x-ing a POC doesn't actually get you 10x velocity). MVP, though? No way. Today's frontier models are nowhere near smart enough to write a non-trivial product (i.e. something that others are meant to use), minimal or otherwise, without careful supervision. Anthropic weren't able to get agents to write even a usable C compiler (not a huge deal to begin with), even with a practically infeasible amo…

There is a million things in between a C compiler and a non-trivial product. They do make a ton of horrible architectural decisions, but I only need to review the output/ask questions to guide that, not review every diff.

A C compiler with an existing C compiler as oracle, existing C compilers in the training set, and a formal spec, is already the easiest possible non-trivial product an agent could build without human review.

You could have it build something that takes fewer lines of code, but you aren’t gonna to find much with that level of specification and guardrails.

Re: Over-editing refers to a model modifying code beyond what is necessary

#139

Post by jauntywundrkind »

jstanley wrote: post #4

Conversely, I often find coding agents privileging the existing code when they could do a much better job if they changed it to suit the new requirement. I guess it comes down to how ossified you want your existing code to be. If it's a big production application that's been running for decades then you probably want the minimum possible change. If you're just experimenting with stuff and the project didn't exist at…

To get the agent to think for itself sometimes it feels like I have to delete a bunch of code and markdown first. Instruction to refactor/reconsider broadly has such mild success, I find.

I'll literally run an agent & tell it to clean up a markdown file that has too much design in it, delete the technical material, and/or delete key implementations/interfaces in the source, then tell a new session to do the work, come up with the design. (Then undelete and reconcile with less naive sessions.)

Path dependence is so strong. Right now I do this flow manually but I would very much like to codify this, make a skill for this pattern that serves so well.

Re: Over-editing refers to a model modifying code beyond what is necessary

#140

Post by pron »

andoando wrote: post #121
pron wrote: post #112

Earlier quoted context omitted.

POC, sure (although 10x-ing a POC doesn't actually get you 10x velocity). MVP, though? No way. Today's frontier models are nowhere near smart enough to write a non-trivial product (i.e. something that others are meant to use), minimal or otherwise, without careful supervision. Anthropic weren't able to get agents to write even a usable C compiler (not a huge deal to begin with), even with a practically infeasible amo…

There is a million things in between a C compiler and a non-trivial product. They do make a ton of horrible architectural decisions, but I only need to review the output/ask questions to guide that, not review every diff.

A C compiler is a 10-50KLOC job, which the agents bricked in 0 days despite a full spec and thousands of hand-written tests, tests that the software passed until it collapsed beyond saving. Yes, smaller products will survive longer, but how would you know about the time bombs that agents like hiding in their code without looking? When I review the diffs I see things that, if had let in, the codebase would have died in 6-18 months.

BTW, one tip is to look at the size of the codebase. When you see 100KLOC for a first draft of a C compiler, you know something has gone horribly wrong. I would suggest that you at least compare the number of lines the agent produced to what you think the project should take. If it's more than double, the code is in serious, serious trouble. If it's in the Asking the agent questions is good - as an aid to a review, not as a substitute. The agents lie with a high enough frequency to be a serious problem.

The models don't yet write code anywhere near human quality, so they require much closer supervision than a human programmer.

Post reply on HN