Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Over-editing refers to a model modifying code beyond what is necessary

nrehiew.github.io

Post reply on HN
Print view View original

261–267 of 267 posts

Re: Over-editing refers to a model modifying code beyond what is necessary

#261

Post by Izkata »

cortesoft wrote: post #67
betenoire wrote: post #48

Earlier quoted context omitted.

Said another way, compilers are a translation of existing formal code. Compilers don't add features, they don't create algorithms (unrolling, etc., notwithstanding), they are another expression of the same encoded solution. LLMs are nothing like that

LLMs are just translating text into output, too, and are running on deterministic computers like every other bit of code we run. They aren't magic. It is just the scope that makes it appear non-deterministic to a human looking at it, and it is large enough to be impossible for a human to follow the entire deterministic chain, but that doesn't mean it isn't in the end a function that translates input data into output…

> is just the scope that makes it appear non-deterministic to a human looking at it

They are non-deterministic due to both the GPUs they run on and some additional randomness intentionally added through the "temperature" parameter.

Re: Over-editing refers to a model modifying code beyond what is necessary

#262

Post by alsetmusic »

jampekka wrote: post #224
Lionga wrote: post #223

Earlier quoted context omitted.

also add "no hallucinations" and "make it works this time pretty please" while also say Claude will go to jail if does not do it right should work all the time (so like 60%)

There are of course limits to what prompting can do, but it does steer the models. In TFA they found that prompting mitigates over-editing up to about 10 percentage points.

Similar to the observation (by simonw) that they respond reasonably to "TDD: Red => Green"

I've used that ever since. Works most of the time, but other stuff is often failing and I've learned to become distrustful of an agent very quickly. One mistake where I point it out and the agent corrects itself is fine if it keeps working well after. A second mistake when it's trying to fix the first one or an inability to understand or a claim that it fixed it but it didn't is instant termination (after dumping context for the next agent).

Re: Over-editing refers to a model modifying code beyond what is necessary

#263

Post by thatxliner »

maxbond wrote: post #156
hathawsh wrote: post #142

I'm either in a minority or a silent majority. Claude Code surpasses all my expectations. When it makes a mistake like over-editing, I explain the mistake, it fixes it, and I ask it to record what it learned in the relevant project-specific skills. It rarely makes that mistake again. When the skill file gets big, I ask Claude to clean and compact it. It does a great job. It doesn't really make sense economically for…

When I see people talking about Claude Code becoming "unusable" for them recently, I believe them, but I don't understand. It's a deeply flawed and buggy piece of software but it's very effective. One of the strangest things about AI to me is that everyone seems to have a radically different experience.

You have to try Codex. My friend's been trying to convert me for months and he was right all along: with Codex you don't GSD or whatever prompting metaframework. You rarely (I actually haven't need to do this at all) need to ask it to retry because its implementation is bugged: it literally just works first try.

Maybe that's because the harness maybe (not sure; haven't looked at their source code) has it baked in? Doesn't matter; the point is that it works.

Now, the one thing I heavily dislike is the UI it generates...it doesn't seem to realize that matching UI patterns with the existing codebase is quite important.

Re: Over-editing refers to a model modifying code beyond what is necessary

#264

Post by samusiam »

foo12bar wrote: post #80

I've noticed AI's often try and hide failure by catching exceptions and returning some dummy value maybe with some log message buried in tons of extraneous other log messages. And the logs themselves are often over abbreviated and missing key data to successfully debug what is happening. I suspect AI's learned to do this in order to game the system. Bailing out with an exception is an obvious failure and will be pena…

In my experience this "gaming" behavior is easily caught by just asking another agent (could just be another session of Claude Code) to review the code changes.

Re: Over-editing refers to a model modifying code beyond what is necessary

#265

Post by Calavar »

kowbell wrote: post #258
Calavar wrote: post #216

Earlier quoted context omitted.

It's interesting how variable people's experiences seem to be. Personally, I tend to get crap quality code out of Claude. Very branchy. Very un-DRY. Consistently fails to understand the conventions of my codebase (e.g. keeps hallucinating that my arena allocator zero initializes memory - it does not). And sometimes after a context compaction it goes haywire and starts creating new regressions everywhere. And while yo…

> In terms of limits, I usually find myself hitting the rate limit after two or three requests. I'd absolutely love to see exactly what you're doing (...well, maybe in a world where I had unlimited time or could clone myself...) because as tight as the usage limits are I absolutely cannot fathom hitting them THAT early. What are the requests like, and have you noticed what is Claude doing during them? Is it reading a…

I'm writing a compiler. When I have Claude write a new feature, I have validate that suite against a test suite of ~200 tiny programs.

I have a shell script that automates this. If all tests pass, the shell script prints "200/200 passing" with very little token spend. If only 190/200 pass, the shell script reports the names of every test that failed, and now Claude does a process of

1) run the compiler binary -> 2) get assembly output and inspect for obvious errors -> 3) assemble -> 4) verify that the assembler did not report errors -> 5) run test binary, connect with gdb, and find the issue -> 6) edit the compiler source -> 7) recompile the compiler -> 8) back to 1

multiplied by 10 for the 10 failing tests. This eats up tokens very quickly. I realize that not every use case is going to look like this. But if I didn't have Claude verify against the test suite, then I'd be getting regressions left and right, and then what's the point?

The whole codebase (tests included) is less than 15k lines, so I don't think that's the issue. No MCPs. CLAUDE.md about 1.5k lines.

Re: Over-editing refers to a model modifying code beyond what is necessary

#266

Post by lelanthran »

cortesoft wrote: post #96
lelanthran wrote: post #92

Earlier quoted context omitted.

I think you have an incorrect mental model of how LLMs work. There's plenty of resources online to rectify that, though.

I think you may be misreading my comment, then, because I know how LLMs work. Which part of my comment do you think shows that I don’t?

> I think you may be misreading my comment, then, because I know how LLMs work. Which part of my comment do you think shows that I don’t?

Your repeated comparisons to compilers in the context of determinism display a lack of understanding of compilers, LLMs and determinism. That's quite an impressive trifecta!

Unless you are expecting the definition of determinism to include this: https://news.ycombinator.com/item?id=47911524 ... you can't really be surprised when people call out your definition of determinism as completely and utterly wrong.

A compiler is a function `f(x) -> y`. For given x, a specific compiler will in actual use always result in the specified y. There is no currently used LLM that I am aware of that is able to do that for any non-trivial input.

An LLM is a function `f(x) -> P(y)`. For a given x, you get a probability of y as the output.

If you are still unable to acknowledge that there is difference between `f(x) -> y` and `f(x) -> P(y)`, then there really isn't any reason to continue this thread.

Re: Over-editing refers to a model modifying code beyond what is necessary

#267

Post by Mitch2 »

anonu wrote: post #5

Here, the author means the agent over-edits code. But agents also do "too much": as in they touch multiple files, run tests, do deployments, run smoke tests, etc... And all of this gets abstracted away. On one hand, its incredible. But on the other hand I have deep anxiety over this: 1. I have no real understanding of what is actually happening under the hood. The ease of just accepting a prompt to run some script th…

[flagged]
Post reply on HN