Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

61–70 of 265 posts

Re: There is an AI code review bubble

#61

Post by The_Fox »

candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

I installed CodeRabbit for our reviews in GitLab and am pretty happy with the results, especially considering the low price ($15/user/mo I think).

It regularly finds problems, including subtle but important problems that human reviewers struggle to find. And it can make pretty good suggestions for fixes.

It also regularly complains about things that are possible in theory but impossible in practice, so we've gotten used to just resolving those comments without any action. Maybe if we used types more effectively it would do that less.

We pay a lot more attention to what CodeRabbit says than what DeepSource said when use used it.

Re: There is an AI code review bubble

#62

Post by tayo42 »

shagie wrote: post #22
candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

In some code that I was working on, I had // stuff obj.setSomeData(something); // fifteen lines of other code obj.setSomeData(something); // more stuff The 'something' was a little bit more complex, but it was the same something with slightly different formatting. My linter didn't catch the repeat call. When asking the AI chat for a review of the code changes it did correctly flag that there was a repeat call. It als…

Unit tests catch that kind of stuff

Re: There is an AI code review bubble

#63

Post by justapassenger »

candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

AI code review to me is similar to AI code itself. It's good (and constantly getting better) at dealing with mundane things, like - is the list reversed correctly? Are you dealing with pointers correctly? Do you have off by 1 issues?

Where they suck is high level problems like - is the code actually solving the business problem? Is it using right dependencies? Does it fit into broader design?

Which is expected for me and great help. I'm more happy as a human to spend less time checking if you're managing lifecycle of the pointer correctly and focus on ensuring that code is there to do what it needs to do.

Re: There is an AI code review bubble

#64

Post by anthonypasq96 »

tayo42 wrote: post #62
shagie wrote: post #22

Earlier quoted context omitted.

In some code that I was working on, I had // stuff obj.setSomeData(something); // fifteen lines of other code obj.setSomeData(something); // more stuff The 'something' was a little bit more complex, but it was the same something with slightly different formatting. My linter didn't catch the repeat call. When asking the AI chat for a review of the code changes it did correctly flag that there was a repeat call. It als…

Unit tests catch that kind of stuff

youre verifying std lib function call counts in unit tests? lmao.

Re: There is an AI code review bubble

#65

Post by cmrdporcupine »

"While some other products have built out great UIs for humans to review code in an AI-assisted paradigm, we have chosen to build for what we consider to be an inevitable future - one where code validation requires vanishingly little human participation."

Ok good, now I know not to bother reading through any of their marketing literature, because while the product at first interested me, now I know it's exactly not what I want for my team.

The actual "bubble" we have right now is a situation where people can produce and publish code they don't understand, and where engineers working on a system no longer are forced to reckon with and learn the intricacies of their system, and even senior engineers don't gain literacy into the very thing they're working on, and so are somewhat powerless to assess quality and deal with crisis when it hits.

The agentic coding tools and review tools I want my team (and myself) to have access to are ones that ones that force an explicit knowledge interview & acquisition process during authoring and involve the engineer more intricately in the whole flow.

What we got instead with claude code & friends is a thing way too eager to take over the whole thing. And while it can produce some good results it doesn't produce understandable systems.

To be clear, it's been a long time since writing code has been the hard part of the job? in many many domains. The hard part is systems & architecture and while these tools can help with that, there's nothing more potentially terrifying pthan a team full of people who have agentically produced a codebase that they cannot holistically understand the nuances of.

So, yeah, I want review tools for that scenario. Since these people have marketed themselves off the table... what is out there?

Re: There is an AI code review bubble

#66

Post by cmrdporcupine »

kaishin wrote: post #55

We used Greptile where I work and it was so bad we decided to switch to Claude. And even Claude isn’t nearly as good at reviewing as an experienced programmer with domain knowledge.

My experience is that Claude or others are good at pointing out things I will want to look at and then I can go review more thoroughly. So it's helped to some degree.

But like everything else with it, it tries to do too much.

What I want is a review "wizard" agent -- something that identifies the pieces I should look at, and takes me through them diff by diff asking me to read them, while offering its commentary ("this appears to be XX....") and letting me make my own.

Re: There is an AI code review bubble

#67

Post by fcarraldo »

shagie wrote: post #22
candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

In some code that I was working on, I had // stuff obj.setSomeData(something); // fifteen lines of other code obj.setSomeData(something); // more stuff The 'something' was a little bit more complex, but it was the same something with slightly different formatting. My linter didn't catch the repeat call. When asking the AI chat for a review of the code changes it did correctly flag that there was a repeat call. It als…

Why isn’t `obj` immutable?

Re: There is an AI code review bubble

#68

Post by tayo42 »

anthonypasq96 wrote: post #64
tayo42 wrote: post #62

Earlier quoted context omitted.

Unit tests catch that kind of stuff

youre verifying std lib function call counts in unit tests? lmao.

You can do that with mocks if it's important that something is only called once, or likely there's some unintended side effect of calling it twice and tests woukd catch the bug

Re: There is an AI code review bubble

#70

Post by zmmmmm »

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so human attention is the primary bottleneck here. Thus poor signal to noise ratio isn't a side issue, it's one of the core issues.

As a result, I'm mostly using this selectively so far, and I wouldn't want it turned on by default for every PR.

Post reply on HN