Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

181–190 of 265 posts

Re: There is an AI code review bubble

#181

Post by ljm »

johntash wrote: post #162
eieio wrote: post #144

Earlier quoted context omitted.

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release? I like this review method too though, and like that some pr review tools have a 'suggest changes' and 'apply changes' button now too

I think it's a good idea to have a style guide of sorts that you can point to when people sweat the small stuff.

Re: There is an AI code review bubble

#182

Post by pbalau »

johntash wrote: post #162
eieio wrote: post #144

Earlier quoted context omitted.

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release? I like this review method too though, and like that some pr review tools have a 'suggest changes' and 'apply changes' button now too

> What if you have two people with different ideas of how to name a certain variable and they just flip the name back and forth every release?

Fire both. There is no amount of skill and productivity that can justify that amount of pettiness.

Re: There is an AI code review bubble

#183

Post by [deleted] »

eieio wrote: post #144
Quarrelsome wrote: post #75

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

[deleted]

Re: There is an AI code review bubble

#184

Post by Cthulhu_ »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

This is why you should set guidelines for reviews (like e.g. https://go.dev/wiki/CodeReviewComments), and ideally automate as much as possible. I'm guilty of this as well, leaving loads of nitpicky code style comments - but granted, this was before Prettier was a thing. In hindsight, I could've spent all that time building a code formatter myself lol.

Re: There is an AI code review bubble

#186

Post by catlifeonmars »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

Let me throw something out there: poor naming obscures and distracts from functional issues. You are right about a good reviewer, but a good author strives for clarity in addition to correctness.

As an aside, naming is highly subjective. Like in writing, you tailor naming to the problem domain and the audience.

Re: There is an AI code review bubble

#187

Post by bjackman »

sebra wrote: post #174

I've tried Greptile and it's pretty much pure noise. I ran it for 3 PRs and then gave up. Here are three examples of things it wasted my time on in those 3 PRs: * Suggested to silence exception instead of crash and burn for "style" (the potential exception was handled earlier in code but it did not manage to catch that context). When I commented that silencing the exception could lead to uncaught bugs it replies "You…

My experience is that basic generic agents are useless but an agent with extensive prompting about your usecase is extremely valuable.

In my case using these prompts:

https://github.com/masoncl/review-prompts

Took things from "pure noise" to a world where, if you say there's a bug in your patch, people's first question will be "has the AI looked at it?"

FWIW in my case the AI has never yet found _the_ bug I was hunting for but it has found several _other_ significant bugs. I also ran it against old commits that were already reviewed by excellent engineers and running in prod. It found a major bug that wasn't spotted in human review.

Most of the "noise" I get now just leads me to say "yeah I need to add more context to the commit message". E.g the model will say "you forgot to do X" when X is out of scope for the patch and I'm doing it in a later one. So ideally the commit messages should mention this anyway.

Re: There is an AI code review bubble

#188

Post by Manfred »

Fuzzy automated reviews should always run in an interactive loop with a developer on their workstation and contain enough context to quickly assess if they are valid or not.

When developers create a PR, they already feel they are "done", and they have likely already shifted their focus on another task. False positive are horrible at this point, especially when they keep changing with each push of commits.

Re: There is an AI code review bubble

#189

Post by __0x01 »

Is "AI code review" a correct term?

A code review requires reasoning and understanding, things that to my knowledge a generative model cannot do.

Surely the most an AI code review ever could be is something that looks like a code review.

Re: There is an AI code review bubble

#190

Post by rat9988 »

__0x01 wrote: post #189

Is "AI code review" a correct term? A code review requires reasoning and understanding, things that to my knowledge a generative model cannot do. Surely the most an AI code review ever could be is something that looks like a code review.

Given we are more interested in the end than in the mean, it is a good usage.
Post reply on HN