Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When I reject AI code even if it works

vinibrasil.com

Post reply on HN
Print view View original

181–184 of 184 posts

Re: When I reject AI code even if it works

#181

Post by andrekandre »

codemog wrote: post #27
embedding-shape wrote: post #15

Earlier quoted context omitted.

> There are a lot of workplaces where there isn’t a good mechanism to push back on this and the tech debt just keeps growing. If the "big ball of spaghetti" theory holds, where software companies who can't manage the debt stumble over themselves as they continue to add to the big ball of spaghetti code, I guess we'll see a row of companies declaring "software bankruptcy" or something in some/many months, depending on…

Coding agents have been better than the average "enterprise" programmer for a while now and nobody wants to admit it or talk about it. I have never seen an agent output an implementation called FooImpl that's tens of thousands of LOC in a single file, but I have seen plenty of human code like this. People call coding agents bad because they don't know the asinine meaningless conventions at their particular company wh…

no

i've never seen even a junior do something as crazy as displaying a page sheet ui from literally a color object, yes, a literal color...

Re: When I reject AI code even if it works

#182

Post by jameslaneyno9 »

The sycophancy problem disappears when you stop pushing back on the agent and start having a human review the plan before the agent writes anything. Push back on a human who wrote the plan and their agreement or disagreement will actually be meaningfull. Push back on the agent and you get "you're absolutely right..".

The plan also solves "I can't explain this code" because you wrote the plan before the build, so you can explain it.

After tracking some internal metrics recently we found plan review costs 0.7 hours on average compared to PR review that costs 16 hours. We rejected 13 out of 165 plans meaning no code was written.

The one gap this doesn't close is that the agent drifts from the plan. We run a separate adversarial check that compares the diff against the approved plan and flags anything the plan didn't specify. That catches scope drift without reading every line.

Re: When I reject AI code even if it works

#183

Post by frankish »

cassianoleal wrote: post #146
glimshe wrote: post #145

Earlier quoted context omitted.

Another version of this issue is when you push back but you were NOT "right to push back". In other words, the LLM original solution was better than the pushback. Most of the time my pushbacks are true improvements, but I've seen a couple of instances where the LLM was happy to downgrade their own good solution.

I've had those as well. Sometimes I'm asking clarifying questions because I'm not sure about the solution, and the LLM "interprets" that as pushback (as opposed to curiosity / enquiry), and sycophancy takes over. Sometimes it will simply change the code without ever answering the questions, or it will answer the questions along with it, but incorrectly - or with bad assumptions.

This is why I love plan mode and switch to it as soon as I see the agent starting to deviate from the plan because it hit a wall. Gotta get back to aligning on design decisions, approach, reverting any debug code, etc. before progressing. (Damn, I have more instructions to add to AGENTS.md lol)

Although this was more of an issue when using opusplan as Sonnet loves to do this. I only use Opus now if using Claude.

Post reply on HN