Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

When I reject AI code even if it works

vinibrasil.com

Post reply on HN
Print view View original

61–70 of 184 posts

Re: When I reject AI code even if it works

#61

Post by kajman »

wwind123 wrote: post #36

I use 3 AI's (Claude, GPT and Gemini) to review each other's design plans and implementation on the same code base. Each often catches problems the others miss. I try to make sure the architecture docs of the code base are refreshed regularly based on recent changes, so it's easier for humans and AI agents to make sense of the code. I also regularly stop all other developments and just focus on auditing the code base…

I'm always curious when I see these stories. How long have you been doing this, for what sort of work, and was the codebase mature before you began working like this?

Re: When I reject AI code even if it works

#62

Post by cadamsdotcom »

equinumerous wrote: post #37
cadamsdotcom wrote: post #35

If you reject AI code that works then your mindset is still too hands on. Put another way - you still have some loops to work on taking yourself out of. The agent should’ve delivered code that was acceptable as a first pass. Agents respond really well to feedback! They have no ego and they’ll happily improve code if told where and how. But you need to provide the tools that provide that feedback without your involvem…

I am very curious what some of your lint rules look like in practice. In my mind a lot of the AI-isms in my code that I hate are stylistic or a matter of taste, not necessarily something I could write a deterministic rule to check. But I want to hear more. Like, what kind of linters did you create and which were highest impact?

Start at https://github.com/cadamsdotcom/CodeLeash/blob/main/.pre-com... where you’ll see the custom lints.

Then have a look at https://github.com/cadamsdotcom/CodeLeash/blob/main/scripts/... (which was test-driven alongside https://github.com/cadamsdotcom/CodeLeash/blob/main/tests/un...)

The script can exit 2 to block the agent, and whatever it prints to stderr is shown to the agent. That’s a pretty darn flexible way to enforce whatever you like.

Despite this being in the codebase I still have no idea what python’s ast stuff is or does - I just let the agent rip, ensured it did TDD and reviewed it all to make sure the tests & code looked reasonable. I didn’t write this code and don’t want to. But I’ve watched it catch hundreds of dumb AI-isms, and watched the agent go “okay” and fix them ;) it’s been paying for itself over and over for months :)

Re: When I reject AI code even if it works

#63

Post by pyridines »

resonious wrote: post #16
Aurornis wrote: post #10

Even using Fable (while it was briefly available), having it refine a plan, and directing it to make only small incremental changes, I still found reasons to reject its first pass at a lot of work. There was a lot of “You’re right to push back” responses. A lot of incidents where it would creat some giant complex set of abstractions to accomplish something that I could find ways to do much more elegantly and in a mor…

All Claude models are huge suck ups. The "you're absolutely right" meme is real even if that exact phrase doesn't show up as much anymore. I don't want to start a fight or anything but IME Codex has a bit more of a spine. If you point out something weird, it sometimes gives a good reason for it. Whereas Claude will always say "whoopsie you're right as always sir" even when it's me who missed something.

IME it's Claude that pushes back, and Codex that just does the thing. It's happened once or twice where I've told Claude bluntly and directly "do this" and it responded "no, here's why that's a bad idea..." Maybe it's just my CLAUDE.md.

Not sure if there are sycophancy benchmarks for coding agents

Re: When I reject AI code even if it works

#64

Post by unknownfuture »

cadamsdotcom wrote: post #51
unknownfuture wrote: post #39

Earlier quoted context omitted.

Frankly, if that's truly your flow, then you cannot possibly know if the code really does what you expect it to do. "TDD" isn't some magic trick. The tests codify the expected behavior. But if you don't review them for correctness, if you let the LLM build them blindly, then you have no idea what those tests assert and can make no claims about whether the code then does what you expect. That's fine. That's your choic…

Huh? I design everything with plan mode and review every line. Nothing happens to my codebase that I don’t decide should happen. With my way of working, tech debt doesn’t exist because I never have to create it. You’ve made a bunch of assumptions you’re not conscious of. And now you’re blaming me for that. Open your mind, you never know what you might (un)learn.

So then your response has nothing to do with the post.

The thesis of the post is (paraphrasing): "if an AI wrote it, and I don't immediately grok it or if the code quality is low, I throw it away, even if on the surface it seems to work, because simply 'working' isn't enough to say a piece of code is acceptable."

I'd add as a corollary "and therefore I would never want to be accountable for that code."

If you're reviewing every line then it sounds like you have no argument with the writer and I don't understand what your point is.

Your very first paragraph says:

> If you reject AI code that works then your mindset is still too hands on. Put another way - you still have some loops to work on taking yourself out of.

But if you do indeed "review every line" then you seem pretty damn in the loop yourself and I don't understand what you think taking oneself out of the loop is.

Re: When I reject AI code even if it works

#65

Post by piterrro »

I feel the same way, reading AI built feature entire output makes me cognitively overloaded as well - I can only do so many throughout the day.

What I found myself doing is operating in two modes: 1. For projects that require my attention, I plan and instruct LLM, when needed will draft some code and ask agent to make it better or finish the mundane part (write code and leave gaps with comments asking agent to finish) 2. Full automode where I use spec driven development and TDD - I only ask for changes based on existing PRD, which agent also have to update. Here I do not look at the code at all.

Seems to be working just fine.

Re: When I reject AI code even if it works

#66

Post by littlecosmic »

skydhash wrote: post #22
datadrivenangel wrote: post #4

"The reality is that code that runs and makes the CI green can still be a bad solution, and engineering has always been about implementing adequate, scalable, and extensible solutions." Adequate often means done and cheap

I was just watching a video about system engineering and the following stucks: Stakeholder needs : What people wants to get done with the product Management needs : How to manage the spending of resources (time, money,…) to create the product Engineering needs : What is the product You have to balance the three. Sometimes it’s simple and easy to get right. Sometimes it’s complex enough, you’re never truly sure until…

The stakeholders just want to send emails and excels around, someone in management has a budget for a productivity enhancing tool to replace that and the engineers have a half-baked solution that some sales guys are saying is the second coming.

Re: When I reject AI code even if it works

#67

Post by wwind123 »

jimbobimbo wrote: post #55
wwind123 wrote: post #36

I use 3 AI's (Claude, GPT and Gemini) to review each other's design plans and implementation on the same code base. Each often catches problems the others miss. I try to make sure the architecture docs of the code base are refreshed regularly based on recent changes, so it's easier for humans and AI agents to make sense of the code. I also regularly stop all other developments and just focus on auditing the code base…

This is the way. I use gh copilot and have opus interrogate me and write the plan, then gpt review the plan and provide feedback; repeat this multiple times until gpt is either satisfied or starts to nitpick on unimportant stuff. Then sanity check the plan myself and have gpt implement it. Each implementation is also reviewed by me before merging to master. I complete PRs only when I'm satisfied with the implementati…

Cool. Yeah it's important to have a big picture of the product, to steer the AI's towards the right direction in their work.

Re: When I reject AI code even if it works

#68

Post by tedajax »

lemagedurage wrote: post #48
jdw64 wrote: post #40

Coding with AI eventually comes down to two paths, I've realized. One is using AI exclusively for everything. The other is not using it at all. There is almost no middle ground. The reason is that as the complexity and depth of the problem increase, the code AI generates increasingly follows enterprise level patterns. The deeper the meaning of what I input, the more AI tends to produce code that goes beyond my own ar…

Own the design and let AI write the code. Spend the extra free time on becoming a better/broader architect.

I'd rather blowtorch my nipples off than yell at a computer all day

Re: When I reject AI code even if it works

#70

Post by wwind123 »

kajman wrote: post #61
wwind123 wrote: post #36

I use 3 AI's (Claude, GPT and Gemini) to review each other's design plans and implementation on the same code base. Each often catches problems the others miss. I try to make sure the architecture docs of the code base are refreshed regularly based on recent changes, so it's easier for humans and AI agents to make sense of the code. I also regularly stop all other developments and just focus on auditing the code base…

I'm always curious when I see these stories. How long have you been doing this, for what sort of work, and was the codebase mature before you began working like this?

Yeah, this one is easy: I have been doing this for half a year. I have a couple of projects worked out this way, all green-field projects, code base grew from 0 to tens of thousand of lines each.
Post reply on HN