The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem • Hacker News Board

Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem

#31

Post by avikalp » Sun, Mar 02, 2025, 1:30 AM UTC

shermantanktop wrote: post #10

An AI tool that told the author how to create a CR that was readable and changed the minimum amount of stuff in one go would actually be helpful. Multipage CRs are only ok if it’s a bulk reformat or file move operation.

This is only true for "development branch" CRs/pull requests. The whole is greater than the sum of the parts. Every small change in the feature that you are building might make complete sense, so every dev-to-feature branch pull request would get approved easily.

But if you not also reviewing the feature-to-main branch pull request, you are just inviting problems. That is a bigger CR that you should review carefully, and there is no way that could be a small CR.

Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem

#32

Post by Herring » Mon, Mar 03, 2025, 12:34 AM UTC

avikalp wrote: post #28

Herring wrote: post #6

AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases. It just talks like it's very smart, and humans apparently have a bias for persuasive communication skills. It's also very fast, which humans also think indicates general intelligence. But it's not, and that's why most LLM tools are author-focused, so th…

I have had a similar discussion with a fellow On-Deck Founder, and here is where we reached: - More than being "good enough", it is about taking responsibility. - A human can make more mistakes than an AI, and they are still the more appropriate choice because humans can be held responsible for their actions. AI, by its very nature, cannot be 'held responsible' -- this has been agreed upon based on years of research…

I'd be really hesitant to say anything involving humans and human judgement under uncertainty is trivial. What if the customer wants the car to drive aggressively, maybe speed a little where it "seems" safe? Should the car stop for an object that might be a plastic bag or a child's backpack? Even manual drivers are difficult to "verify" because accidents and traffic violations depend on interpretations of events, which is why we often have to go to court.

Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem

#33

Post by shermantanktop » Mon, Mar 03, 2025, 10:01 AM UTC

avikalp wrote: post #31

shermantanktop wrote: post #10

An AI tool that told the author how to create a CR that was readable and changed the minimum amount of stuff in one go would actually be helpful. Multipage CRs are only ok if it’s a bulk reformat or file move operation.

This is only true for "development branch" CRs/pull requests. The whole is greater than the sum of the parts. Every small change in the feature that you are building might make complete sense, so every dev-to-feature branch pull request would get approved easily. But if you not also reviewing the feature-to-main branch pull request, you are just inviting problems. That is a bigger CR that you should review carefully,…

I’ve done that gig many times. I take your point but if there’s a problem, figuring out who to blame always means going back to the original dev CR.

Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem

#34

Post by CompoundEyes » Thu, Mar 06, 2025, 12:40 PM UTC

donmcronald wrote: post #21

CompoundEyes wrote: post #17

Earlier quoted context omitted.

Sorry I meant that it's javascript / typescript so we can deterministically orchestrate a series of prompts and shape their output exactly how we'd like. Returning the review as structured output as a JSON object is very helpful for this. If the review result seems bungled, run a judge prompt at the end and tell it to go try again ^_^.

That makes sense. You could even run against multiple models or future models then, right? I can see some value in that because maybe 2 years from now the models will be able to surface issues that weren’t detected originally. I suppose you could run against the whole codebase in the future, but could also imagine something that could track down where a bug was introduced. Do you save the reviews or discard them?

Not presently saving the reviews but we could go back and export the DIFF text for two commit hashes with git to recreate what was reviewed in a CI pipeline. This is also handy for tweaking the prompt towards what you're expecting to see on local machine.

I agree the script should be versioned because the output of the one model to the next varies so much and the potential that even an updated version of one model could break it. Treating model versions like an npm package dependency almost.

Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem

#35

Post by CompoundEyes » Thu, Mar 06, 2025, 12:46 PM UTC

HoyaSaxa wrote: post #23

CompoundEyes wrote: post #12

I put in a code reviewer that runs and comments when a pull request is created using Github actions and Microsoft GenAIScript. It's pretty straightforward. The key thing is we have total control over the prompt to fit our repo and devs needs, can make it multi-stage and deterministic using Typescript code or use agents in GenAIScript to open adjacent files for more context. The value we've received is that a dev can…

I’d be interested in seeing the scripts if you are able to share (redacted) versions of them

I can't do that at the moment but the GenAIScript project repo has a good plug and play version that I built upon. In my opinion agent Typescript classes that return structured data tied to interfaces/types with their own memory is where it's at. Also been experimenting with a state machine class to hot potato the output between an agent that judges the results and the worker agent until satisfied.