Benchmarking GPT-5 on 400 real-world code reviews

Post by rowanG077 » Fri, Aug 08, 2025, 8:31 AM UTC

BlindEyeHalo wrote: post #50

LauraMedia wrote: post #43

Earlier quoted context omitted.

Am I missing something? If LLM-1 is supposed to judge LLM-2, doesn't LLM-1 have to be better than LLM-2? If LLM-1 is only 40% as good at coding as LLM-2, why would you trust the LLM with the lesser knowledge?

At the heart of the P vs NP problem lies the observation that solution verification seems to be much easier than solution generation. If that applies in this context is another question but I think it is not unreasonable to assume that the judge needs to be less powerful than the performer. Or in other words, I don't need to be a chef myself to decide if a meal is good or not.

Post by jama211 » Fri, Aug 08, 2025, 8:42 AM UTC

rowanG077 wrote: post #51

BlindEyeHalo wrote: post #50

Earlier quoted context omitted.

At the heart of the P vs NP problem lies the observation that solution verification seems to be much easier than solution generation. If that applies in this context is another question but I think it is not unreasonable to assume that the judge needs to be less powerful than the performer. Or in other words, I don't need to be a chef myself to decide if a meal is good or not.

That really doesn't hold for all problems. You can imagine any number of problems where a valid solution is easier, complexity wise, to generate than it is to validate. A trivial example is semiprime factorization. Easy to generate any semiprime, hard to factor.

Post by [deleted] » Fri, Aug 08, 2025, 8:48 AM UTC

rowanG077 wrote: post #51

BlindEyeHalo wrote: post #50

Earlier quoted context omitted.

At the heart of the P vs NP problem lies the observation that solution verification seems to be much easier than solution generation. If that applies in this context is another question but I think it is not unreasonable to assume that the judge needs to be less powerful than the performer. Or in other words, I don't need to be a chef myself to decide if a meal is good or not.

That really doesn't hold for all problems. You can imagine any number of problems where a valid solution is easier, complexity wise, to generate than it is to validate. A trivial example is semiprime factorization. Easy to generate any semiprime, hard to factor.

Post by fcantournet » Fri, Aug 08, 2025, 8:50 AM UTC

Leherenn wrote: post #34

shinycode wrote: post #8

I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?

Only has a sanity check/better hints. But I use it for my own PRs, not others'. Usually it's not much to review and easy to agree/disagree with. I haven't found it to be really useful so far, but it's also very little added work, so for now I keep on using it. If it saves my ass even just once, it will probably be worth it overall.

Post by jacquesm » Fri, Aug 08, 2025, 9:08 AM UTC

jeltz wrote: post #58

jacquesm wrote: post #52

Earlier quoted context omitted.

That response is quite in line with the typical human based PR response on a first draft. There is a possibility that machine based PR reviews are better: for instance because they are not prejudiced based on who is the initiator of the PR and because they don't take other environmental factors into account. You'd expect a machine to be more neutral, so on that front the machine should and possibly could score better…

I wouldn't necessarily expect a machine to be more neutral. Machines can easily be biased too.