Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

161–170 of 265 posts

Re: There is an AI code review bubble

#161

Post by solid_fuel »

spooky_action wrote: post #112
Quarrelsome wrote: post #102

Earlier quoted context omitted.

> Naming comments can be very useful in code that gets read by a lot of people. It can make the process of understanding the code much quicker. yes but it can be severely diminishing returns. Like lets step back a second and ask ourselves if: var itemCount = items.Count; vs var numberOfItems = items.Count; is ever worth spending the time discussing, versus how much of a soft improvement it makes to the code base. I'v…

Sorry for the dumb question, is the second version actually better than the first? Because I prefer the first. But perhaps you chose this as a particularly annoying/unuseful comment

If I was going to nitpick it I would point out that `itemsCount` could easily be confused with `items.Count`, or vice versa, depending on syntax highlighting. That kind of bug can have a negative impact if one or the other is mutated while the function is running.

So clearly distinguishing the local `numberOfItems` from `items.Count` _could_ be helpful. But I wouldn't ping it in a review.

Re: There is an AI code review bubble

#162

Post by johntash »

eieio wrote: post #144
Quarrelsome wrote: post #75

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release?

I like this review method too though, and like that some pr review tools have a 'suggest changes' and 'apply changes' button now too

Re: There is an AI code review bubble

#163

Post by materialpoint »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

How is that different from today's SA, like CodeQL and SonarQube? Most of the feedback is just sh*t and drives programmers towards making senseless perfections that just double the amount of work had to be done later to toggle or tune behaviour, because the configurable variables are gone due to bad static code analysis. Clearly present intent and convience like: Making a method virtual, adding a public method, not making a method static when it is likely to use instance fields in the future --- these good practices are shunned in all SA just because the rules are opportunistic, not real.

Re: There is an AI code review bubble

#164

Post by dakshgupta »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

The signal-to-noise ratio problem is unexpectedly difficult.

We wrote about our approach to it some time ago here - https://www.greptile.com/blog/make-llms-shut-up

Much has changed on our approach since then, so we'll probably write a a new blog post.

The tl;dr of what makes it hard is - different people have different ideas of what a nitpick is - it's not a spectrum, the differences are qualitative - LLMs are reluctant to risk downplaying the severity of an issue and therefore are unable to usefully filter out nits. - theory: they are paid by the token and so they say more stuff

Re: There is an AI code review bubble

#165

Post by shakna »

materialpoint wrote: post #163
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

How is that different from today's SA, like CodeQL and SonarQube? Most of the feedback is just sh*t and drives programmers towards making senseless perfections that just double the amount of work had to be done later to toggle or tune behaviour, because the configurable variables are gone due to bad static code analysis. Clearly present intent and convience like: Making a method virtual, adding a public method, not m…

My experience at work: Claude regularly says to use one method over another, because it's "safer"... But the method doesn't actually exist in that language. Seems to get rather confused between C# and C++, despite also getting told the language, before and after getting handed the code.

Re: There is an AI code review bubble

#166

Post by brabel »

SchemaLoad wrote: post #137
zenolijo wrote: post #83

Earlier quoted context omitted.

Naming comments can be very useful in code that gets read by a lot of people. It can make the process of understanding the code much quicker. On the other hand, if it's less important code or the renaming is not clearly an improvement it can be quite useless. But I've met some developers who has the opinion of reviews as pointless and just say "this works, just approve it already" which can be very frustrating when i…

A lot of these comments are not pointing out actual issues, just "That's not how I would have done it" type comments.

And the most amazing part is that we got a mini PR review in the comments to a single line of code someone posted just to show an example of useless debates :D

Re: There is an AI code review bubble

#167

Post by brabel »

zeroCalories wrote: post #118
causalscience wrote: post #91

Earlier quoted context omitted.

Yeah or worse like my boss. We don't have a style guide. But he always wants style changes in every PR, and those style changes are some times contradictory across different PRs. Eventually I've told him "if your comment does not affect performance or business logic, I'm ignoring it". He finally got the message. The fact that he accepted this tells me that deep down he knew his comments were just bike shedding.

You should have a style guide, or adopt one. Having uniform code is incredibly valuable as it greatly reduces the cognitive load of reading it. Same reason that Go's verbose "err != nil" works so well.

Style guidelines should be enforced automatically. Leaving that for humans to verify is a recipe for conflict and frustration.

Re: There is an AI code review bubble

#168

Post by brabel »

xmprt wrote: post #98
Quarrelsome wrote: post #75

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

Human comments tend to be short and sweet like "nit: rename creatorOfWidgets to widgetFactory". Whereas AI code review comments are long winded not as precise. So even if there are 20 humans comments, I can easily see which are important and which aren't.

We are using BitBucket at work and decided to turn on RovoDev as reviewer. It absolutely doesn’t do that. Few but relevant comments are the norm and when we don’t like something it says we tell it in its instructions file to stop doing that. It has been working great!

Re: There is an AI code review bubble

#169

Post by coopykins »

simbleau wrote: post #107

After testing several bots in our org, specifically Devin, Graphite, and Cursor, I’ve noticed Cursor is the better bug bot out there right now.

Same here, tested a bunch and cursor has been given little noise and usually decent suggestions. In this case its on a react app, so other projects might not find it as good.

Re: There is an AI code review bubble

#170

Post by knes »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

At augment code we specifically build our code review tool to find noise to signal ratio problem. In benchmark our comments are 2 to 3x more likely to get fixed compared to bugbot coderabbit etc

You should check it at Augmentcode.com

Post reply on HN