Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

221–230 of 265 posts

Re: There is an AI code review bubble

#221

Post by jolt42 »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

Don't get me started on not CamelCasing acronyms, acronyms aren't more important than regular words! :)

Re: There is an AI code review bubble

#222

Post by storystarling »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

I suspect the noise is largely an artifact of cost optimization. Most tools restrict the context to just the diff to save on input tokens, rather than traversing the full dependency graph. Without seeing the actual definitions or call sites, the model is forced to speculate on side effects.

Re: There is an AI code review bubble

#223

Post by [deleted] »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

[deleted]

Re: There is an AI code review bubble

#224

Post by sebra »

Dylan-CodeRab wrote: post #220
sebra wrote: post #174

I've tried Greptile and it's pretty much pure noise. I ran it for 3 PRs and then gave up. Here are three examples of things it wasted my time on in those 3 PRs: * Suggested to silence exception instead of crash and burn for "style" (the potential exception was handled earlier in code but it did not manage to catch that context). When I commented that silencing the exception could lead to uncaught bugs it replies "You…

I am a member of the CodeRabbit tech support team, would you be able to provide me the ticket number you have open with us? I'd be happy to get this escalated internally so we can get this resolved for you ASAP.

Thanks Dylan. Turns out my colleague actually had a teams call with you yesterday and the issue was confirmed and prioritised on your end.

You have a great product. Looking forward to get back to using it!

Re: There is an AI code review bubble

#225

Post by jcgl »

gbalduzzi wrote: post #219
jcgl wrote: post #210

Earlier quoted context omitted.

That sounds great. Was that proprietary tooling? I'd be interested in some such thing.

If I understood correctly, the same can be done on VS Code with the github plugins (for github PRs) It's pretty straightforward: you checkout a PR, move around, and either make some edits (that you can commit and push to the feature branch) or add comments.

Good to know about its existence. I think I'll have to do my own sleuthing though, since I'm a (neo)vim user who dislikes GitHub.

Re: There is an AI code review bubble

#226

Post by jv22222 »

We built an internal code review tool at the day job and are getting pretty good results with it (CLI tool).

Here's a summary of the top-level ideas behind it. Hope it's helpful!

Core Philosophy

- "Advisor, not gatekeeper" - Every issue includes a "Could be wrong if..." caveat because context matters and AI can't see everything. Developers make the final call.

(Just this idea makes it less annoying and stops devs going down rabbit holes because it it pretty good at thinking why it might be wrong)

- Prompt it to be critical but not pedantic - Focus on REAL problems that matter (bugs, security, performance), not style nitpicks that linters handle.

- Get the team to run it on the command line just before each commit. Small, focused reviews not after batching 10 commits. Small diffs get better feedback.

Smart Context Gathering

- Full file contents, not just diffs - The tool reads complete changed files plus 1-level-deep imports to understand how changed code interacts with the codebase.

Prompt Engineering

- Diff-first, context-second - The diff is marked as "REVIEW THIS" while context files are explicitly marked "DO NOT REVIEW - FOR UNDERSTANDING ONLY" to prevent false positives on unchanged code. BUT that extra context makes a huge difference in correctness.

- Structured output format - Emoji-prefixed bullets ( Critical, Major, Minor), max 3 issues per section, no fluff or praise.

- Explicit "Do NOT" list - Prevents common AI review mistakes: don't flag formatting (Prettier handles it), don't flag TypeScript errors (IDE shows them), don't repeat issues across files, don't guess line numbers.

Final note

- Also plugged it in to a github action for last pass, but again non blocking.

Re: There is an AI code review bubble

#227

Post by zeroCalories »

brabel wrote: post #167
zeroCalories wrote: post #118

Earlier quoted context omitted.

You should have a style guide, or adopt one. Having uniform code is incredibly valuable as it greatly reduces the cognitive load of reading it. Same reason that Go's verbose "err != nil" works so well.

Style guidelines should be enforced automatically. Leaving that for humans to verify is a recipe for conflict and frustration.

Ideally yes, but there's plenty of cases where that's not desirable or possible.

For example, most people would agree you should use exhaustive checks when possible(matching in rust, records in typescript, etc.). But how do you enforce that? Ban other types of control flow? But even if you find a good balanced way to enforce it, you won't always want to enforce it. There's plenty of good use cases where you explicitly don't want a check to be exhaustive. At which point now you gotta make sure there's an escape mechanism to whatever crackhead check you've setup. Better to just leave a comment with a link to your style guide explaining why this is done. Many experienced developers that are new to rust or typescript simply never think of things like this, so it's worthwhile to document it.

Re: There is an AI code review bubble

#228

Post by sotix »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

I follow a consistent comment pattern[0] that makes blocking vs non-blocking easy to identify.

[0]: https://conventionalcomments.org/

Re: There is an AI code review bubble

#229

Post by PurelyApplied »

eieio wrote: post #144
Quarrelsome wrote: post #75

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

If minor mistakes are corrected without the PR author's input, do they ever learn to stop making those mistakes themselves? It seems like a system where you just never bother to learn, e.g., style conventions, because reviewers just apply edits as needed.
Post reply on HN