Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Over-editing refers to a model modifying code beyond what is necessary

nrehiew.github.io

Post reply on HN
Print view View original

91–100 of 267 posts

Re: Over-editing refers to a model modifying code beyond what is necessary

#91

Post by btbuildem »

I wish there was a reliable way to choke the agents back and prevent them from doing this. Every line of code added is a potential bug, and they overzealously spew pages and pages of code. I've routinely gone through my (hobby) projects and (yes, still with the aid of an LLM) trimmed some 80% of the generated code with barely any loss of functionality.

The cynic in me thinks it's done on purpose to burn more tokens. The pragmatist however just wants full control over the harness and system prompts. I'm sure this could be done away with if we had access to all the knobs and levers.

Re: Over-editing refers to a model modifying code beyond what is necessary

#92

Post by lelanthran »

cortesoft wrote: post #59
mathieudombrock wrote: post #41

Earlier quoted context omitted.

A major difference is that _someone_ knew what was going on (compiler devs).

That is an interesting difference, I agree. Although, while the compiler devs might know what was going on in the compiler, they wouldn't know what the compiler was doing with that particular bit of code that the FORTRAN developer was writing. They couldn't possibly foresee every possible code path that a developer might traverse with the code they wrote. In some ways, you could say LLMs are like that, too; the LLM d…

I think you have an incorrect mental model of how LLMs work.

There's plenty of resources online to rectify that, though.

Re: Over-editing refers to a model modifying code beyond what is necessary

#93

Post by Lihh27 »

ok_dad wrote: post #10
anonu wrote: post #5

Here, the author means the agent over-edits code. But agents also do "too much": as in they touch multiple files, run tests, do deployments, run smoke tests, etc... And all of this gets abstracted away. On one hand, its incredible. But on the other hand I have deep anxiety over this: 1. I have no real understanding of what is actually happening under the hood. The ease of just accepting a prompt to run some script th…

Why are you letting the LLM drive? Don't turn on auto-approve, approve every command the agent runs. Don't let it make design or architecture decisions, you choose how it is built and you TELL that clanker what's what! No joke, if you treat the AI like a tool then you'll get more mileage out of it. You won't get 10x gains, but you will still understand the code.

That's the trap though. The moment you approve every step, you're no longer getting the product that was sold to you. You're doing code review on a stochastic intern. The whole 10x story depends on you eventually looking away.

Re: Over-editing refers to a model modifying code beyond what is necessary

#94

Post by brianwmunz »

I feel like a core of this is that agents aren't exactly a replacement for a junior developer like some people say. A junior dev has its own biases, predispositions, history and understanding of the internal and external aspects of a product and company. An AI agent wants to do what you ask in the best way possible which is...not always what a dev wants :) The fix the article talks about is simple but shows that these models have no inherent sense of project scope or proportionality. You have to give context (as much context as possible) explicitly to fill in the gaps so it infers less and makes smaller decisions.

Re: Over-editing refers to a model modifying code beyond what is necessary

#96

Post by cortesoft »

lelanthran wrote: post #92
cortesoft wrote: post #59

Earlier quoted context omitted.

That is an interesting difference, I agree. Although, while the compiler devs might know what was going on in the compiler, they wouldn't know what the compiler was doing with that particular bit of code that the FORTRAN developer was writing. They couldn't possibly foresee every possible code path that a developer might traverse with the code they wrote. In some ways, you could say LLMs are like that, too; the LLM d…

I think you have an incorrect mental model of how LLMs work. There's plenty of resources online to rectify that, though.

I think you may be misreading my comment, then, because I know how LLMs work. Which part of my comment do you think shows that I don’t?

Re: Over-editing refers to a model modifying code beyond what is necessary

#98

Post by betenoire »

cortesoft wrote: post #67
betenoire wrote: post #48

Earlier quoted context omitted.

Said another way, compilers are a translation of existing formal code. Compilers don't add features, they don't create algorithms (unrolling, etc., notwithstanding), they are another expression of the same encoded solution. LLMs are nothing like that

LLMs are just translating text into output, too, and are running on deterministic computers like every other bit of code we run. They aren't magic. It is just the scope that makes it appear non-deterministic to a human looking at it, and it is large enough to be impossible for a human to follow the entire deterministic chain, but that doesn't mean it isn't in the end a function that translates input data into output…

just text !== syntactically correct code that solves a defined problem

There is a world of difference between translation and generation. It's even in the name: generative AI. I didn't say anything about magic.

Re: Over-editing refers to a model modifying code beyond what is necessary

#99

Post by cortesoft »

lelanthran wrote: post #90
cortesoft wrote: post #63

Earlier quoted context omitted.

LLMs are deterministic, too. I know there is randomness in the choosing tokens, but that randomness is derived from a random seed that can be repeated.

If I understand your argument, you're saying that models can be deterministic, right? Care to point to any that are set up to be deterministic? Did you ever stop to think about why no one can get any use out of a model with temp set to zero?

No, LLMs ARE deterministic, just like all computer programs are.

I get why that is in practice different then the manner in which compilers are deterministic, but my point is the difference isnt because of determinism.

Re: Over-editing refers to a model modifying code beyond what is necessary

#100

Post by exe34 »

pavel_lishin wrote: post #78
arjie wrote: post #77

Earlier quoted context omitted.

I have never found any utility in that. After all, you can still just review the diffs and ask it for explanation for sections instead.

> After all, you can still just review the diffs anonu has explicitly said that they've wiped a database twice as a result of agents doing stuff. What sort of diff would help against an agent running commands, without your approval?

Hah I run my agent inside a docker with just the code. Anything clever it tries to do just goes nowhere.
Post reply on HN