The main problem with current AI reviewers isn't catching bugs, it's shutting up when there is no bug. Humans have an intuitive filter like "this code is weird, but it works and won't break prod, so I'll let it slide". LLMs lack this, they generate 20 comments about variable naming and 1 comment about a critical race condition. As a result the developer gets fatigue and ignores everything. Until AI learns to understand the context of importance, not just code context, it will remain an expensive linter
There is an AI code review bubble
191–200 of 265 posts
Re: There is an AI code review bubble
#192I still think any business that is based on someone else's model is worthless. I know I'm sounding like the 'dropbox is just FTP' guy, but it really feels like that any good idea will just be copied by OpenAI and Anthropic. If AI code review is proven a good idea is there any reason to expect Codex or Claude Code to not implement some commands to do code review?
The thing is, the "Dropbox is just FTP" guy should be right most of the time when you are selling to experts.
There is no reason to not just ask Claude for a review and then distill this into PR comments. Especially because "every LLM output has to be filtered through a human" is a good policy with the current state of these tools.
However, this industry loves distilling frivolities into web tools and it sells for some unfathomable reason. It is the same with the existing static analyzers etc that some orgs pay for. I do not understand why.
Re: There is an AI code review bubble
#193Earlier quoted context omitted.
Yeah or worse like my boss. We don't have a style guide. But he always wants style changes in every PR, and those style changes are some times contradictory across different PRs. Eventually I've told him "if your comment does not affect performance or business logic, I'm ignoring it". He finally got the message. The fact that he accepted this tells me that deep down he knew his comments were just bike shedding.
I've been in teams like this - people who are lower on the chain of power get run in circles as they change to appease one, then change to appease another then change to go back to appease the first again. Then, going through their code, they make excuses about their code not meeting the same standards they demand. As the other responder recommends, a style guide is ideal, you can even create an unofficial one and po…
> Then, going through their code, they make excuses about their code not meeting the same standards they demand.
Yes!! Exactly. When it comes to my PRs, he once made this snarky comment about him having high expectations in terms of code quality. When it comes to his PRs, he does the things he tells me not to do. In fact, I once sent him a "dis u?" with a link to his own code, as a response to something he told me I shouldn't do. To his credit he didn't make excuses, he responded "I could've done better there, agreed".
In general he's not bad, but his nitpicking is bad. I don't really understand what's going on in his mind that drives this behavior, it's weird.
Re: There is an AI code review bubble
#194My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…
> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…
If you are nitpicking style or conventions that do not have rules in your linting tools, then those should automatically be non-issues, IMO.
Re: There is an AI code review bubble
#195Hot take: Code review is an anti-pattern. We spend a ton of time looking at the code and blocking merges, and the end result is still full of bugs. AI code review only provides a minor improvement. The only reason we do code review at all is humans don't trust that the code works. Know another way to tell if code works? Running it. If our code is so utterly inconceivable that we can't make tests that can accurately a…
Running the code checks if it works now, whereas code review checks if it will work in a year and if anyone else can understand it.
Tests don't catch architectural mistakes or time bombs. If you remove reviews and rely solely on tests, you end up with a "working" big ball of mud that is impossible to maintain. AI won't help if it's the one generating the mud.
Re: There is an AI code review bubble
#196I can't get over how every single code rabbit ad was some incorrectly classified bug / completely wrong to begin with or pointless at best.
Re: There is an AI code review bubble
#197Earlier quoted context omitted.
> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…
at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…
I just this morning had someone "nitpick" on a PR I made and ask for a change that would have broken the code.
If the reviewer can make changes without someone reviewing their change, it's just waiting to blow up in. your face.
Re: There is an AI code review bubble
#198Earlier quoted context omitted.
it "nit" short for nitpick? I think prefixing PR comments with prefixes like that is very helpful for dealing with this problem.
Yes, but I don't know how effective it is. 99% of the time someone leaves a 'nit' the other person fixes it. So we're still dealing with most of them like regular comments. Only once or twice I've been like "nah, I like my way better" but I can only do that if they also leave an LGTM. Sometimes they do. There's one or two people that will hold your code hostage until you reply to every little nit. At that point they…
I wonder if there's a psychological benefit though. If someone states up front that they know something is just a nitpick, the author might be less likely to push back, and therefore it's less likely to end up in a bike shedding back-and-forth.
Re: There is an AI code review bubble
#199Is "AI code review" a correct term? A code review requires reasoning and understanding, things that to my knowledge a generative model cannot do. Surely the most an AI code review ever could be is something that looks like a code review.
Given we are more interested in the end than in the mean, it is a good usage.
Please can you elaborate? We are more intersted in "the end" in what sense?
Re: There is an AI code review bubble
#200Earlier quoted context omitted.
How would you suggest tests around: void func() { printEvens(someCall().stream().filter(n -> n % 2 == 0).toList()); } void printEvens(List nums) { nums.stream().filter(n -> n % 2 == 0).forEach(n -> System.out.println(n)); } The first filter is redundant in this example. Duplicate code checkers are checking for exactly matching lines. I am unaware of any linter or static analyzer that would flag this. What's more, uni…
Idk how exactly to do it in cpp becasue I'm not familiar with the tooling You could write a test that makes sure the output of someCall is passed directly to printeven without being modified. The example as you wrote is hard to test in general. It's probably not something you would write if your serious about testing.
> You could write a test that makes sure the output of someCall is passed directly to printeven without being modified.
But why would anyone ever do that? There's nothing incorrect about the code, it's just less efficient than it should be. There's no reason to limit calls to printEven to accept only output from someCall.