Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

There is an AI code review bubble

greptile.com

Post reply on HN
Print view View original

81–90 of 265 posts

Re: There is an AI code review bubble

#81

Post by estimator7292 »

mulmboy wrote: post #40
dakshgupta wrote: post #10

Earlier quoted context omitted.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

People more often say that to save face by implying the issue you identified would be reasonable for the author to miss because it's subtle or tricky or whatever. It's often a proxy for embarrassment

When mature, funtional adults say it, the read is "wow, I would have missed that, good job, you did better than me".

Reading embarrassment into that is extremely childish and disrespectful.

Re: There is an AI code review bubble

#82

Post by marginalia_nu »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

That's not even mentioning a not insignificant part of the point of code reviews is to propagate understanding of the evolution of the code base among other team members. The reviewer benefits from the act of reviewing as well.

Re: There is an AI code review bubble

#83

Post by zenolijo »

Quarrelsome wrote: post #75
zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

Naming comments can be very useful in code that gets read by a lot of people. It can make the process of understanding the code much quicker.

On the other hand, if it's less important code or the renaming is not clearly an improvement it can be quite useless. But I've met some developers who has the opinion of reviews as pointless and just say "this works, just approve it already" which can be very frustrating when it's a codebase with a lot of collaboration.

Re: There is an AI code review bubble

#84

Post by thesurlydev »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

For the signal to noise reason, I start with Claude Code reviewing a PR. Then I selectively choose what I want to bubble up to the actual review. Often times, there's additional context not available to the model or it's just nit picky.

Re: There is an AI code review bubble

#85

Post by anthonypasq96 »

noitpmeder wrote: post #69
anthonypasq96 wrote: post #64

Earlier quoted context omitted.

youre verifying std lib function call counts in unit tests? lmao.

You're not verifying the observable behavior of your application? lmao

what happened to not testing implementation details?

Re: There is an AI code review bubble

#86

Post by anthonypasq96 »

tayo42 wrote: post #68
anthonypasq96 wrote: post #64

Earlier quoted context omitted.

youre verifying std lib function call counts in unit tests? lmao.

You can do that with mocks if it's important that something is only called once, or likely there's some unintended side effect of calling it twice and tests woukd catch the bug

i know you could do it, im asking why on earth you would feel its vital to verify stream.filter() was called twice in a function

Re: There is an AI code review bubble

#87

Post by RetpolineDrama »

candiddevmike wrote: post #5

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

>The SOTA isn't capable of using a code diff as a jumping off point.

The low quality of HN comments has been blowing my mind.

I have quite literally been doing what you describe every working day for the last 6+ months.

Re: There is an AI code review bubble

#88

Post by heliumtera »

segmondy wrote: post #74

If you give LLM a hammer everything looks like a nail, you give it a saw everything looks like wood. You ask LLM to find issues, it will find "issues" At the end of the day, you will have to fix those issues, if you decide to have another LLM fix those issues, by the time you are done with that cycle, you are going to end up with code that will be thoroughly over engineered.

If by engineering you mean doing whatever vibes you feel, than yeah, over engineering.

If by engineering you mean using the engineering design process than it would not be engineered at all, let alone over engineered.

Re: There is an AI code review bubble

#89

Post by biophysboy »

zmmmmm wrote: post #70

My experience with using AI tools for code review is that they do find critical bugs (from my retrospective analysis, maybe 80% of the time), but the signal to noise ratio is poor. It's really hard to get it not to tell you 20 highly speculative reasons why the code is problematic along with the one critical error. And in almost all cases, sufficient human attention would also have identified the critical bug - so hu…

I absolutely hate the verbosity of AI. I know that you can give it context; I have done it, and it helps a little. It will still give me 10 "ideas", many of which are closely related to each other.

Re: There is an AI code review bubble

#90

Post by quietbritishjim »

noitpmeder wrote: post #69
anthonypasq96 wrote: post #64

Earlier quoted context omitted.

youre verifying std lib function call counts in unit tests? lmao.

You're not verifying the observable behavior of your application? lmao

A redundant filter() isn't observable (except in execution time).

You could pick it up if you were to explicitly track whether it's being called redundantly but it'd be very hard and by the time you'd thought of doing that you'd certainly have already manually checked the code for it.

Post reply on HN