Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

11–20 of 265 posts

Re: There is an AI code review bubble

#11

Post by liamconnell »

dakshgupta wrote: post #4
personjerry wrote: post #2

I don't really understand how this differentiates against the competition. > Independence Any "agent" running against code review instead of code generation is "independent"? > Autonomy Most other code review tools can also be automated and integrated. > Loops You can also ping other code review tools for more reviews... I feel like this article actually works against you by presenting the problem and inadequately so…

> Independence It is, but when a model/harness/tools/system prompts are the same/similar in the generator and reviewer fail in similar ways. Question: Would you trust a Cursor review of Claude-written code more, less, or the same as a Cursor review of Cursor-written code? > Autonomy Plenty of tools have invested heavily in AI-assisted review - creating great UIs to help human reviewers understand and check diffs. Our…

> It is, but when a model/harness/tools/system prompts are the same/similar in the generator and reviewer fail in similar ways.

Is there empirical evidence for that? Where is it on an epistemic meter between (1) “it sounds good when I say it”, and (10) “someone ran evaluation and got significant support.”

“Vibes” (2/3 on scale) are ok, just honestly curious.

Re: There is an AI code review bubble

#12

Post by ahmadyan »

trjordan wrote: post #3

1. I absolutely agree there's a bubble. Everybody is shipping a code review agent. 2. What on earth is this defense of their product? I could see so many arguments for why their code reviewer is the best, and this contains none of them. More broadly, though, if you've gotten to the point where you're relying on AI code review to catch bugs, you've lost the plot. The point of a PR is to share knowledge and to catch st…

> 2. What on earth is this defense of their product?

i think the distribution channel is the only defensive moat in low-to-mid-complexity fast-to-implement features like code-review agents. So in case of linear and cursor-bugbot it make a lot of sense. I wonder when Github/Gitlab/Atlassian or Xcode will release their own review agent.

Re: There is an AI code review bubble

#13

Post by dakshgupta »

quanwinn wrote: post #7

I liked that the post is self-aware that it's promoting its own product. But the writing seemed more focus on the philosophy behind code reviews and the impact of AI, and less on the mechanics of how greptile differs from competitors. I was hoping to see more on the latter.

Thanks! We go over that on many other pages. Here are some:

https://www.greptile.com/benchmarks https://www.greptile.com/greptile-vs-coderabbit https://www.greptile.com/greptile-vs-bugbot

Re: There is an AI code review bubble

#14

Post by geooff_ »

This article has a catchy headline, but there's really no content to it. This is content marketing without content. It seems like every week on Hacker News, there's a dozen of these. All seemingly code reviewers, too. Keep it to LinkedIn.

Re: There is an AI code review bubble

#16

Post by sdenton4 »

dakshgupta wrote: post #4
personjerry wrote: post #2

I don't really understand how this differentiates against the competition. > Independence Any "agent" running against code review instead of code generation is "independent"? > Autonomy Most other code review tools can also be automated and integrated. > Loops You can also ping other code review tools for more reviews... I feel like this article actually works against you by presenting the problem and inadequately so…

> Independence It is, but when a model/harness/tools/system prompts are the same/similar in the generator and reviewer fail in similar ways. Question: Would you trust a Cursor review of Claude-written code more, less, or the same as a Cursor review of Cursor-written code? > Autonomy Plenty of tools have invested heavily in AI-assisted review - creating great UIs to help human reviewers understand and check diffs. Our…

Independence is ridiculous - the underlying llm models are too similar on their training days and methodologies to be anything like independent. Trying different models may somewhat reduce the dependency, but all have read stack overflow, Reddit, and GitHub in their training.

It might be an interesting time to double down on automatically building and checking deterministic models of code which were previously too much of a pain to bother with. Eg, adding type checking to lazy python code. These types of checks really are model independent, and using agents to build and manage them might bring a lot of value.

Re: There is an AI code review bubble

#17

Post by onedognight »

dakshgupta wrote: post #10
candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

I fully agree. Claude’s review comments have been 50% useful, which is great. For comparison I have almost never found a useful TeamScale comment (classic static analyzer). Even more important, half of Claude’s good finds are orthogonal to those found by other human reviewers on our team. I.e. it points out things human reviewers miss consistently and v.v.

Re: There is an AI code review bubble

#18

Post by aurareturn »

sastraxi wrote: post #9

Contrary to some of the other anecdotes in this thread, I've found automated code review to discover some tricky stuff that humans missed. We use https://www.cubic.dev/

Before I push any code, I always ask 2 different frontier LLMs to review the changes for any potential issues. Saved my ass a few times before pushing to production.

Re: There is an AI code review bubble

#19

Post by tadfisher »

dakshgupta wrote: post #10
candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

Not trying to sidetrack, but a figure like that is data, not evidence. At the very minimum you need context which allows for interpretation; 9,078 positive author comments would be less impressive if Greptile made 1,000,000 comments in that time period, for example.

Re: There is an AI code review bubble

#20

Post by lenerdenator »

trjordan wrote: post #3

1. I absolutely agree there's a bubble. Everybody is shipping a code review agent. 2. What on earth is this defense of their product? I could see so many arguments for why their code reviewer is the best, and this contains none of them. More broadly, though, if you've gotten to the point where you're relying on AI code review to catch bugs, you've lost the plot. The point of a PR is to share knowledge and to catch st…

> More broadly, though, if you've gotten to the point where you're relying on AI code review to catch bugs, you've lost the plot.

> The point of a PR is to share knowledge and to catch structural gaps.

Well, it was to share knowledge and to catch structural gaps.

Now you have an idea, for better or for worse, that software needs to be developed AI-first. That's great for the creation of new code but as we all know, it's almost guaranteed that you'll get some bad output from the AI that you used to generate the code, and since it can generate code very fast, you have a lot of it to go through, especially if you're working on a monorepo that wasn't architected particularly well when it was written years ago.

PRs seem like an almost natural place to do this. The alternative is the industry finding a more appropriate place to do this sort of thing in the SDLC, which is gonna take time, seeing as how agentic loop software development is so new.

Post reply on HN