Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

41–50 of 265 posts

Re: There is an AI code review bubble

#41

Post by realusername »

shagie wrote: post #22
candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

In some code that I was working on, I had // stuff obj.setSomeData(something); // fifteen lines of other code obj.setSomeData(something); // more stuff The 'something' was a little bit more complex, but it was the same something with slightly different formatting. My linter didn't catch the repeat call. When asking the AI chat for a review of the code changes it did correctly flag that there was a repeat call. It als…

I had a similar review by AI except my equivalent of setSomeData was stateful and needed to be there in both places, the AI just didn't understand any of it.

Re: There is an AI code review bubble

#42

Post by gherkinnn »

candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

Opus 4.5 catches all sorts of things a linter would not, and with little manual prompting at that. Missing DB indexes, forgotten migration scenarios, inconsistencies with similar services, an overlooked edge case.

Now I'm getting a robot to review the branch at regular intervals and poking holes in my thinking. The trick is not to use an LLM as a confirmation machine.

It doesn't replace a human reviewer.

I don't see the point of paying for yet another CI integration doing LLM code review.

Re: There is an AI code review bubble

#45

Post by james_marks »

realusername wrote: post #41
shagie wrote: post #22

Earlier quoted context omitted.

In some code that I was working on, I had // stuff obj.setSomeData(something); // fifteen lines of other code obj.setSomeData(something); // more stuff The 'something' was a little bit more complex, but it was the same something with slightly different formatting. My linter didn't catch the repeat call. When asking the AI chat for a review of the code changes it did correctly flag that there was a repeat call. It als…

I had a similar review by AI except my equivalent of setSomeData was stateful and needed to be there in both places, the AI just didn't understand any of it.

When this happens to me it makes me question my design.

If the AI doesn’t understand it, chances are it’s counter-intuitive. Of course not all LLM’s are equal, etc, etc.

Re: There is an AI code review bubble

#47

Post by maxverse »

themafia wrote: post #35

> Unfortunately, code review performance is ephemeral and subjective > Today's agents are better than the median human code reviewer Which is it? You cannot have it both ways.

> Today's agents are better than the median human code reviewer

"...at catching issues and enforcing standards, and they're only getting better".

I took this to mean what good code review is is subjective. But if you clearly define standards and patterns for your code, your linter/automated tools/ AI code reviewer will always catch more than humans.

Re: There is an AI code review bubble

#48

Post by Sharlin »

onedognight wrote: post #17
dakshgupta wrote: post #10

Earlier quoted context omitted.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

I fully agree. Claude’s review comments have been 50% useful, which is great . For comparison I have almost never found a useful TeamScale comment (classic static analyzer). Even more important, half of Claude’s good finds are orthogonal to those found by other human reviewers on our team. I.e. it points out things human reviewers miss consistently and v.v.

TBH that sounds like TeamScale just has too verbose default settings. On the other hand, people generally find almost all of the lints in Clippy's [1] default set useful, but if you enable "pedantic" lints, the signal-to-noise ratio starts getting worse – those generally require a more fine-grained setup, disabling and enabling individual lints to suit your needs.

[1] https://doc.rust-lang.org/stable/clippy/

Re: There is an AI code review bubble

#49

Post by maxverse »

Maybe I'm buying into the cool-aid, but I actually really liked the self-aware tone of this post.

> Based on our benchmarks, we are uniquely good at catching bugs. However, if all company blogs are to be trusted, this is something we have in common with every other AI code review product. One just has to try a few, and pick the one that feels the best.

Post reply on HN