Live data from Hacker News

There is an AI code review bubble

greptile.com

201–210 of 265 posts

Re: There is an AI code review bubble

#201
post #144

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

Was that Jane Street? I remember watching a presentation from someone there about such a system.

If not, any chance this tooling is openly available?

Re: There is an AI code review bubble

#202
post #197
post #144

Earlier quoted context omitted.

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

I just this morning had someone "nitpick" on a PR I made and ask for a change that would have broken the code. If the reviewer can make changes without someone reviewing their change, it's just waiting to blow up in. your face.

Yes, in the system I'm describing if a reviewer changed your code, you reviewed their change.

Re: There is an AI code review bubble

#203
post #144

Earlier quoted context omitted.

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

Was that Jane Street? I remember watching a presentation from someone there about such a system. If not, any chance this tooling is openly available?

> Was that Jane Street?

yep

Re: There is an AI code review bubble

#204

None of these tools perform particularly well and all lack context to actually provide a meaningful review beyond what a linter would find, IMO. The SOTA isn't capable of using a code diff as a jumping off point. Also the system prompts for some of them are kinda funny in a hopelessly naive aspirational way. We should all aspire to live and breathe the code review system prompt on a daily basis.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

For it to be evidence, you would need to know the number of Greptile comments made and how many of those comments were instead considered to be poor. You need to contrast false positive rate with true positive rate to simply plot a single point along a classifier curve. You would then need to contrast that with a control group of experts or a static linter which means you would need to modify the "conservativeness" of the classifier to produce multiple points along its ROC curve, then you could compare whether the classifier is better or worse than your control by comparing the ROC curves.

Sample number of true positives says more or less nothing on its own.

Re: There is an AI code review bubble

#205

Earlier quoted context omitted.

Yes, but I don't know how effective it is. 99% of the time someone leaves a 'nit' the other person fixes it. So we're still dealing with most of them like regular comments. Only once or twice I've been like "nah, I like my way better" but I can only do that if they also leave an LGTM. Sometimes they do. There's one or two people that will hold your code hostage until you reply to every little nit. At that point they…

I wonder if there's a psychological benefit though. If someone states up front that they know something is just a nitpick, the author might be less likely to push back, and therefore it's less likely to end up in a bike shedding back-and-forth.

This and when an author wants to ignore it, they do. You don't need to justify your choice since the person is openly saying "I'm bikeshedding" to you.

Re: There is an AI code review bubble

#206

Earlier quoted context omitted.

I agree that none perform _super_ well. I would argue they go far beyond linters now, which was perhaps not true even nine months ago. To the degree you consider this to be evidence, in the last 7 days, the authors of a PR has replied to a Greptile comment with "great catch", "good catch", etc. 9,078 times.

Not trying to sidetrack, but a figure like that is data, not evidence. At the very minimum you need context which allows for interpretation; 9,078 positive author comments would be less impressive if Greptile made 1,000,000 comments in that time period, for example.

Bro stop trying to deflate the boosters, they got wares to sell and shares to dump.

Re: There is an AI code review bubble

#207
post #144

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

This is the workflow I've always dreamed of. In a lot of ways making a change which is then submitted as patch to their patch isn't really that different from submitting a comment to their patch. The workflow of doing that directly in editor is just wonderful.

If I had to pick, I actually think ONLY being able to submit "counter-patches" would be better than only being able to submit comments. Comments could just be actual programming language style comments submitted as code changes.

Re: There is an AI code review bubble

#208

Earlier quoted context omitted.

I personally don't give a shit either way but I've worked in dev shops with a clear preference for the second one. I can see their point because the code as natural language parses better but I don't think its strong enough to care about. Sort of place that is fussy about test naming so where I would do smth like: TestSearchCriteriaWhere they'd want Test_That_Where_Clauses_In_Search_Criteria_Work I think its a waste…

Let's take it up a notch! var itemCount = items.Count; depends on what `items` is, no? Is the `.Count` O(1)? Do you really need a variable or is it fine for the (JIT) compiler to take care of it? Is it O(n) and n is significant enough? Maybe you need a variable and spend time arguing about that name. Yes I chose this because almost everyone I know at least would argue you always have to create the variable (and then…

It matters if there's a lot of churn or the test fails a lot but if its a test that I write and for whatever reason it never fails I think we've just wasted our time on being fussy.

I appreciate neither of those test names are great but it was just a strawman example to show the fussiness.

Re: There is an AI code review bubble

#209
post #155
post #134

Earlier quoted context omitted.

Trying to write the easiest code that I could test... I don't think I can without writing an excessively brittle test that would break at the slightest implementation change. So you've got this Java: public List someCall() { return IntStream.range(1,10).boxed().toList(); } public List filterEvens(List ints) { return ints.stream() .filter(i -> i % 2 == 0) .toList(); } int aMethod() { List data = someCall(); return fil…

You mention the tools you can use to make it happen. I think we're at the point where you need concrete examples to talk about whether it's worth it or not. If you have functions that can't be called twice, then you have no other option to test details in the implementation like that. Yeah there's a tradeoff between torturing your code to make everything about it testable and enforce certain behavior or keeping it si…

In functions that you write, that might be possible.

How would you assert that a given std::vector only was filtered by std::ranges::copy_if once? And how would you test that the code that was in the predicate for it wasn't duplicated?

How would you write a failing test for this function keeping the constraint that you are working with std::vector?

    std::vector doThing(const std::vector& nums) {
        std::vector tmp1;
        std::vector tmp2;
        std::ranges::copy_if(data,
                             std::back_inserter(tmp1),
                             [](int n) { return n % 2 == 0; }
        );
        std::ranges::copy_if(tmp1,
                             std::back_inserter(tmp2),
                             [](int n) { return n % 2 == 0; }
        );
        return tmp2;
    }

Re: There is an AI code review bubble

#210
post #144

Earlier quoted context omitted.

> but the signal to noise ratio is poor Nail on the head. Every time I've seen it applied, its awful at this. However this is the one thing I loathe in human reviews as well, where people are leaving twenty comments about naming and then the actual FUNCTIONAL issue is just inside all of that mess. A good code reviewer knows how to just drop all the things that irk them and hyperfocus on what matters, if there's a fun…

at my last job code review was done directly in your editor (with tooling to show you diffs as well). What this meant was that instead of leaving nitpicky comments, people would just change things that were nitpicky but clear improvements. They'd only leave comments (which blocked release) for stuff that was interesting enough to discuss. This was typically a big shock for new hires who were used to the "comment for…

That sounds great. Was that proprietary tooling? I'd be interested in some such thing.
Post reply on HN