Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Over-editing refers to a model modifying code beyond what is necessary

nrehiew.github.io

Post reply on HN
Print view View original

121–130 of 267 posts

Re: Over-editing refers to a model modifying code beyond what is necessary

#121

Post by andoando »

pron wrote: post #112
andoando wrote: post #52

Earlier quoted context omitted.

Because its SO much faster not to have to do all that. I think 10x is no joke, and if you're doing MVP, its just not worth the mental effort.

POC, sure (although 10x-ing a POC doesn't actually get you 10x velocity). MVP, though? No way. Today's frontier models are nowhere near smart enough to write a non-trivial product (i.e. something that others are meant to use), minimal or otherwise, without careful supervision. Anthropic weren't able to get agents to write even a usable C compiler (not a huge deal to begin with), even with a practically infeasible amo…

There is a million things in between a C compiler and a non-trivial product. They do make a ton of horrible architectural decisions, but I only need to review the output/ask questions to guide that, not review every diff.

Re: Over-editing refers to a model modifying code beyond what is necessary

#122

Post by scotty79 »

This seems like something that should be easy to prevent in pi harness. Just tell it to make an extension that before calling file edit tool asks the model to make sure that no lines unconnected with the current topic are going to be unnecessarily changed by this edit.

Re: Over-editing refers to a model modifying code beyond what is necessary

#123

Post by Supermancho »

cortesoft wrote: post #63
ManuelKiessling wrote: post #36

Earlier quoted context omitted.

(I‘m saying this as someone who uses AI for coding a lot and mostly love it) Yeah, but is that really the same? Compilers work deterministically — if it works once, it will work always. LLMs are a different story for now.

LLMs are deterministic, too. I know there is randomness in the choosing tokens, but that randomness is derived from a random seed that can be repeated.

Only if the seed is known. Determinism is often predicated on perfect information. Many programs do not have that. Their operations cannot be reproduced practically. The difference between saying deterministic and non-deterministic is contextual based on if you are concerned with theory or practicality.

Re: Over-editing refers to a model modifying code beyond what is necessary

#124

Post by Supermancho »

cortesoft wrote: post #61
mnkypete wrote: post #38

Earlier quoted context omitted.

Except that compilers are (at least to a large degree) deterministic. It's complexity that you don't need to worry about. You don't need to review the generated assembly. You absolutely need to review AI generated code.

At the end of the day, LLMs are also deterministic. They are running on computers just like all software, and if you have all the same data and random seeds, and you give the same prompt to the same LLM, you will get back the exact same response.

> you give the same prompt to the same LLM, you will get back the exact same response.

Demonstrably incorrect. This is because the model selection, among other data, is not fixed for (I would say most) LLMs. They are constantly changing. I think you meant something more like an LLM with a fixed configuration. Maybe additional constraints, depending on the specific implementation.

Re: Over-editing refers to a model modifying code beyond what is necessary

#125

Post by sornaensis »

ok_dad wrote: post #10
anonu wrote: post #5

Here, the author means the agent over-edits code. But agents also do "too much": as in they touch multiple files, run tests, do deployments, run smoke tests, etc... And all of this gets abstracted away. On one hand, its incredible. But on the other hand I have deep anxiety over this: 1. I have no real understanding of what is actually happening under the hood. The ease of just accepting a prompt to run some script th…

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

I define tools that perform individual tasks, like build the application, run the tests, access project management tools with task context, web search, edit files in the workspace, read only vs write access source control, etc.

The agent only has access to exactly what it needs, be it an implementation agent, analysis agent, or review agent.

Makes it very easy to stay in command without having to sit and approve tons of random things the agent wants to do.

I do not allow bash or any kind of shell. I don't want to have to figure out what some random python script it's made up is supposed to do all the time.

Re: Over-editing refers to a model modifying code beyond what is necessary

#126

Post by zero_shift »

harikb wrote: post #21
anonu wrote: post #5

Here, the author means the agent over-edits code. But agents also do "too much": as in they touch multiple files, run tests, do deployments, run smoke tests, etc... And all of this gets abstracted away. On one hand, its incredible. But on the other hand I have deep anxiety over this: 1. I have no real understanding of what is actually happening under the hood. The ease of just accepting a prompt to run some script th…

On the credentials point. Here is what I find. Day 1: Carefully handles the creds, gives me a lecture (without asking) about why .env should be in .gitignore and why I should edit .env and not hand over the creds to it. Day 2: I ask for a repeat, has lost track of that skill or setting, frantically searches my entire disk, reads .env including many other files, understands that it is holding a token, manually creates…

I think it is just because they are having to load shed! Some days you may be getting much less compute - the main way "thinking" operates, is to just iterate on the result a few more times

Re: Over-editing refers to a model modifying code beyond what is necessary

#127

Post by xboxnolifes »

raincole wrote: post #35
eterm wrote: post #3

It's funny, because the wisdom that was often taught ( but essentially never practiced ) was "Refactor as you go". The idea being that if you're working in an area, you should refactor and tidy it up and clean up "tech debt" while there. In practice, it was seldom done, and here we have LLMs actually doing it, and we're realising the drawbacks.

Really? I've never heard it's considered wise to put refactoring and new features (or bugfixes) in the same commit. Everyone I know from every place I've seen consider it bad. From harmful to a straight rejection in code review. "Refactor-as-you-go" means to refactor right after you add features / fix bugs, not like what the agent does in this article.

Notice how they didn't say to put it in the same commit. The real issue, and why refactor as you go isn't done as much, is the overhead of splitting changes that touch the same code into different commits without disrupting your workflow. It's not as easy as it should be to support this strategy.

Instead you to do it later, and then never do it.

Re: Over-editing refers to a model modifying code beyond what is necessary

#128

Post by ok_dad »

sornaensis wrote: post #125
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

I define tools that perform individual tasks, like build the application, run the tests, access project management tools with task context, web search, edit files in the workspace, read only vs write access source control, etc. The agent only has access to exactly what it needs, be it an implementation agent, analysis agent, or review agent. Makes it very easy to stay in command without having to sit and approve tons…

This is a cool idea, can you write more about how your tools work or maybe short descriptions of a few of them? I’m interested in more rails for my bots.

Re: Over-editing refers to a model modifying code beyond what is necessary

#129

Post by ok_dad »

Lihh27 wrote: post #93
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

That's the trap though. The moment you approve every step, you're no longer getting the product that was sold to you. You're doing code review on a stochastic intern. The whole 10x story depends on you eventually looking away.

Just don’t buy the tools for 10x improvements, buy them for the 1.1x improvement and the help it gives with the annoying stuff like refactoring arguments to a function that’s used all over, writing tests, etc. They can also help reduce cognitive load in certain ways when you just use them to ask about your large code base.

Re: Over-editing refers to a model modifying code beyond what is necessary

#130

Post by ok_dad »

threatofrain wrote: post #72
ok_dad wrote: post #10

Earlier quoted context omitted.

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

Because the degree to which the LLM prompts you back to the terminal is too frequent for the human to engage in parallel work.

I’m basically saying don’t do parallel work, use it as a tool. Just sit there and watch it do stuff, make sure it’s doing what you want, and stop it if it’s doing too much or not what you want to do.

Maybe I’m just weird (actually that’s a given) but I don’t mind babysitting the clanker while it works.

Post reply on HN