The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
1–10 of 35 posts
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#2Is the purpose of these tools really to spend less time? I think their main value is reducing mistakes through having one extra set of eyes, even if mechanical ones, looking at the code.
As a sole developer of a non-trivial open source project, I've recently started using CodeRabbit, very skeptical about it, but right on the first PR, it actually found a bug that my CI tests did not catch, decided to keep it after that.
Gemini Code Assist on the other hand, the first suggestion it did would actually lead to a bug, so that was out immediately.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#3The thing is that inserting AI into the code reviewer side doesn't make too much sense. Unless they have a different AI doing the reviewing than the one that helped to write it, there won't be anything left to say at that stage. The AI was already involved in writing it and as they mention in the article there's points in the writing-with-AI process where the AI editor will try to catch bugs, educate the developer, and so forth. If the reviewing AI can catch further bugs that's just proof the writing AI needs to be tightened up, not that there's a role for a reviewing AI.
The commentary given above is invalid if due to the preferences of the human developers or just weird protocol in their working relationship they end up with different AI's in the two instances. But I think in the long term equilibrium this point applies.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#4We didn't purchase a tool, but instead built our own.
> most AI code review tools on the market today are fundamentally author-focused, not reviewer-focused.
This pretty much describes our experience. Our engineers create a PR and now wait for the review bot to provide feedback. The author will any fix any actual issues the bot brings up and only then will they publish the PR to the rest of the team.
From our experience there are 4 things that make the bot valuable:
1. Any general logical issues in the code are caught with relative certainty (not evaluating a variable value properly or missing a potential edge case, etc).
2. Some of the comments the bot leaves are about the business logic in code and asking about it and having the author provide a clearer explanation helps reviewers to understand what's going on as well if it wasn't clear enough from the code itself.
3. We provide a frontend platform to other engineers in the company that our operations teams interact with. The engineers rarely implement more than 1-2 features a year. We gave the bot a list of coding and frontend guidelines that we enforce (capitalisation rules, title formatting, component spacing, etc) and it will remind reviewers about these requirements.
4. We told it to randomly change it's way of talking from Yoda to Dr Seuss and some of the comments, while correct on a technical level, are absolutely hilarious and can give you a short giggle in an otherwise stressful day.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#5Man, I hate when I'm reading a blogpost and then I realize the whole thing is just an ad for a startup.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#6AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases.
It just talks like it's very smart, and humans apparently have a bias for persuasive communication skills. It's also very fast, which humans also think indicates general intelligence. But it's not, and that's why most LLM tools are author-focused, so that a human expert can catch errors.
The way you know fully autonomous driving is nowhere near ready is by noticing we don't even trust robots to do fully autonomous cooking and cleaning. Similarly, let's see it understand and refactor a massive codebase first.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#7Man, I hate when I'm reading a blogpost and then I realize the whole thing is just an ad for a startup.
One thing I have liked on one of the small business reddits is they require "I will not promote" in submission and as a rule.
Stealth ads are annoying.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#8AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases. It just talks like it's very smart, and humans apparently have a bias for persuasive communication skills. It's also very fast, which humans also think indicates general intelligence. But it's not, and that's why most LLM tools are author-focused, so th…
>AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases.
Ai is better than humans at all those things. It's not good at those things when the context it needs to look over is more than a few thousand tokens.
Rejoice programmer, for your inability to write modular code saved your job.
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#9AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases. It just talks like it's very smart, and humans apparently have a bias for persuasive communication skills. It's also very fast, which humans also think indicates general intelligence. But it's not, and that's why most LLM tools are author-focused, so th…
>AI is not good enough yet for anything requiring deep reasoning, mission-critical work, error detection at a human-expert level, or handling unpredictable edge cases. Ai is better than humans at all those things. It's not good at those things when the context it needs to look over is more than a few thousand tokens. Rejoice programmer, for your inability to write modular code saved your job.
I thought my 3000 line kitchen sink function which mutates globals, uses n+1 fetching, and supports 50 feature flags was a bad idea…maybe not?
Re: The AI Code Review Disconnect: Why Your Tools Aren't Solving Your Real Problem
#10An AI tool that told the author how to create a CR that was readable and changed the minimum amount of stuff in one go would actually be helpful. Multipage CRs are only ok if it’s a bulk reformat or file move operation.