Live data from Hacker News

When I reject AI code even if it works

vinibrasil.com

51–60 of 184 posts

Re: When I reject AI code even if it works

#51

If you reject AI code that works then your mindset is still too hands on. Put another way - you still have some loops to work on taking yourself out of. The agent should’ve delivered code that was acceptable as a first pass. Agents respond really well to feedback! They have no ego and they’ll happily improve code if told where and how. But you need to provide the tools that provide that feedback without your involvem…

Frankly, if that's truly your flow, then you cannot possibly know if the code really does what you expect it to do. "TDD" isn't some magic trick. The tests codify the expected behavior. But if you don't review them for correctness, if you let the LLM build them blindly, then you have no idea what those tests assert and can make no claims about whether the code then does what you expect. That's fine. That's your choic…

Huh?

I design everything with plan mode and review every line. Nothing happens to my codebase that I don’t decide should happen. With my way of working, tech debt doesn’t exist because I never have to create it.

You’ve made a bunch of assumptions you’re not conscious of. And now you’re blaming me for that.

Open your mind, you never know what you might (un)learn.

Re: When I reject AI code even if it works

#54
post #27

Earlier quoted context omitted.

Coding agents have been better than the average "enterprise" programmer for a while now and nobody wants to admit it or talk about it. I have never seen an agent output an implementation called FooImpl that's tens of thousands of LOC in a single file, but I have seen plenty of human code like this. People call coding agents bad because they don't know the asinine meaningless conventions at their particular company wh…

> I have never seen an agent output an implementation called FooImpl that's tens of thousands of LOC in a single file, but I have seen plenty of human code like this. And how long does it take a coding agent to output a thousand lines of code versus a human? The worst human at any company was rate limited by themselves. Those 'average enterprise' programmers aren't going away, they're the ones now spending tens of th…

Which is why one of the big problems for the field right now is that a) most code bases still need someone more skilled than a mere robot driver, and b) many developers are not better than that.

In the past, a team of five mid devs and one good one would be fine, because that good one would ride herd on the mid ones. But now those mid ones are slamming out robot code that they're incapable of meaningfully reviewing (because it's better than they are already), and they're just overwhelming the good dev's capacity.

The solution, of course, is to fire them all -- they're worthless now -- but this is not going to happen quickly, and it's probably for the best that it doesn't.

Re: When I reject AI code even if it works

#55

I use 3 AI's (Claude, GPT and Gemini) to review each other's design plans and implementation on the same code base. Each often catches problems the others miss. I try to make sure the architecture docs of the code base are refreshed regularly based on recent changes, so it's easier for humans and AI agents to make sense of the code. I also regularly stop all other developments and just focus on auditing the code base…

This is the way. I use gh copilot and have opus interrogate me and write the plan, then gpt review the plan and provide feedback; repeat this multiple times until gpt is either satisfied or starts to nitpick on unimportant stuff. Then sanity check the plan myself and have gpt implement it.

Each implementation is also reviewed by me before merging to master. I complete PRs only when I'm satisfied with the implementation, my feedback is addressed, and I fully understand what is going on. Agents are the replacement for typing and productivity multipliers.

I have big picture view of the product, each plan implements only a part of it, scoped to avoid merging unreviwed slop. Probably slower, but result is much better.

Re: When I reject AI code even if it works

#57
post #25

Even using Fable (while it was briefly available), having it refine a plan, and directing it to make only small incremental changes, I still found reasons to reject its first pass at a lot of work. There was a lot of “You’re right to push back” responses. A lot of incidents where it would creat some giant complex set of abstractions to accomplish something that I could find ways to do much more elegantly and in a mor…

These "You're right to push back" scenarios are scary for me. I mostly code ML implementations, and some of the errors Claude Code (CC - have only used Opus 4.7) makes are very sneaky, and if you don't have sufficient experience in the area (I see this with people entering ML and writing their implementations with CC), you wouldn't know when to question CC and will let errors or future pitfalls silently slip into you…

The leakage problem is so pervasive. None of the frontier models seem to have any idea how to actually hold out rows. God help you if you decide to change the data mix.

I was working on creating a next-n-actions predictor for one of our use cases and not paying much attention for a PoC. I was fairly happy with the progress for a few days, before actually reading the eval code and seeing that we leaked the final state in every eval.

It's nice to let claude run loose on porting from framework to framework (port my code from TRL to NemoRL to Tinker to VeRL) but looking at what it does in the intermediate steps makes me want to claw my eyes out. And getting it to adhere to our domain model (e.g. we have an SFTConfig and a .to_trl(), or a Row and a .to_harmony()) is impossible.

Re: When I reject AI code even if it works

#58

I use 3 AI's (Claude, GPT and Gemini) to review each other's design plans and implementation on the same code base. Each often catches problems the others miss. I try to make sure the architecture docs of the code base are refreshed regularly based on recent changes, so it's easier for humans and AI agents to make sense of the code. I also regularly stop all other developments and just focus on auditing the code base…

You’ve transitioned from “individual contributor” to “manager”! (;->

Haha, true!

I do sometimes have to steer the discussions between the AI's to the right direction, if they deviate too far away from the real problem, either because they miss some context, or because my original description of the problem was misleading.

To do that formally, I have a mechanism built-in the review loop where if a comment on a github issue or PR is signed as "-- Human Reviewer", then all AI agents have to treat the comment as the highest priority item to address.

Re: When I reject AI code even if it works

#59
post #3

Earlier quoted context omitted.

According to the author's intention, it is the code that he cannot understand or control. Even if the solution provided by the AI works, he will not adopt it. This is unless he can understand or control it. This should be an assumption. However, if AI provides a solution, as the person using AI, one should conduct research before making a decision. This is not in conflict with or hindered by the use of the ideas prov…

I think this policy is probably more prescriptive than I would go with myself. I like to think of my risk tolerance first to help make that determination. For example, I use a vibecoded internal tool written in Go. I don’t even know how to write Go. Haven’t read a single line of the code. I just wanted to move from bash scripts to using cloud SDKs for performance reasons. But the internal tool is a convenience tool,…

> I just wanted to move from bash scripts to using cloud SDKs for performance reasons.

What is this supposed to mean? How is a “cloud sdk” more performant than a shell script?

Post reply on HN