Benchmarking GPT-5 on 400 real-world code reviews • Hacker News Board

Re: Benchmarking GPT-5 on 400 real-world code reviews

#81

Post by rowanG077 » Fri, Aug 08, 2025, 1:07 PM UTC

mcphage wrote: post #79

rowanG077 wrote: post #51

Earlier quoted context omitted.

That really doesn't hold for all problems. You can imagine any number of problems where a valid solution is easier, complexity wise, to generate than it is to validate. A trivial example is semiprime factorization. Easy to generate any semiprime, hard to factor.

> That really doesn't hold for all problems. But it does hold for this problem.

How so? Asking LLMs to solve a problem can be a problem of any form. For example I just asked this.

Can you give me a very large semiprime?

And claude opus answered:

Here's a very large semiprime:

N = 29927402397991286489627837734179186385188296382227646249397073654051914085318503794952624411151858464246403027505634195232053330357484129331920822220662818816547063469215394303721576869467659309978113411955550111870966028627418736664

This is a over 200-digit semiprime. Factoring semiprimes of this size is computationally intensive, which is why they form the basis of RSA encryption security.

---

Verifying whether this answer is correct is very hard, much harder than generating it.

Problems of this form come up very often. Not even in formal mathematics. Some magic number in the code that you need to reverse engineer to tell it's correct. Some library which you don't have the documentation for but was available when it was written. Hidden intentions or even requirements that are not clear from the code itself. If a weaker LLM is validating a stronger LLM the weaker LLM will simply not grasp the subtleties the stronger LLM created in it's answer. In fact it's a pretty common statement that writing code is easier than reading it. Which is precisely about generation vs validation.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#82

Post by gsandahl » Fri, Aug 08, 2025, 1:22 PM UTC

We are running task specific benchmarks across a number of categories (agentic tasks, context tasks, normalization tasks etc), and on our benchmarks we see Gpt-5 rating slightly below o3. But at a much lower cost.

See https://opper.ai/models

Re: Benchmarking GPT-5 on 400 real-world code reviews

#83

Post by gsandahl » Fri, Aug 08, 2025, 1:25 PM UTC

gsandahl wrote: post #82

We are running task specific benchmarks across a number of categories (agentic tasks, context tasks, normalization tasks etc), and on our benchmarks we see Gpt-5 rating slightly below o3. But at a much lower cost. See https://opper.ai/models

Most of the tasks have assessed with ground truth, occasionally helped with an LLM as a judge to assess the answer if the answer is a sentence and not an exact result.

Example: Given a long travel journal How many cities does the author mention? GPT-5: 12 Expected: 17

Re: Benchmarking GPT-5 on 400 real-world code reviews

#84

Post by david_allison » Fri, Aug 08, 2025, 1:52 PM UTC

rowanG077 wrote: post #81

mcphage wrote: post #79

Earlier quoted context omitted.

> That really doesn't hold for all problems. But it does hold for this problem.

How so? Asking LLMs to solve a problem can be a problem of any form. For example I just asked this. Can you give me a very large semiprime? And claude opus answered: Here's a very large semiprime: N = 299274023979912864896278377341791863851882963822276462493970736540519140853185037949526244111518584642464030275056341952320533303574841293319208222206628188165470634692153943037215768694676593099781134119555501118709660…

> Factoring semiprimes of this size is computationally intensive, which is why they form the basis of RSA encryption security.

Not if it's divisible by 2.

    from sympy import isprime
    num = 29927402397991286489627837734179186385188296382227646249397073654051914085318503794952624411151858464246403027505634195232053330357484129331920822220662818816547063469215394303721576869467659309978113411955550111870966028627418736664
    print(num//2) # 14963701198995643244813918867089593192594148191113823124698536827025957042659251897476312205575929232123201513752817097616026665178742064665960411110331409408273531734607697151860788434733829654989056705977775055935483014313709368332
    print(isprime(num//2)) # False

Re: Benchmarking GPT-5 on 400 real-world code reviews

#85

Post by gen220 » Fri, Aug 08, 2025, 2:05 PM UTC

shinycode wrote: post #8

I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?

Disclosure: my current employer has a product in this space (graphite.dev)

IME the highest value (at the moment) is having an LLM integrated into the PR page, that reads your code + CI log, and effectively operates as a sanity check / semantic linter.

A common workflow for us: is Draft PR -> Passes CI (inclusive of an LLM 'review') -> Published -> Passes Human review -> Scheduled to merge

The goal is to get a higher margin of confidence that your code (1) will not blow up in production (2) faithfully does what it's trying to do.

The value of the LLM reviewer is maybe 80% in the first bucket and 20% in the second bucket, IME. It often catches bugs like "off by one" and "you meant this to be `if not x`, based on the flag name and behavior, not `if x`".

Re: Benchmarking GPT-5 on 400 real-world code reviews

#86

Post by rowanG077 » Fri, Aug 08, 2025, 2:25 PM UTC

david_allison wrote: post #84

rowanG077 wrote: post #81

Earlier quoted context omitted.

How so? Asking LLMs to solve a problem can be a problem of any form. For example I just asked this. Can you give me a very large semiprime? And claude opus answered: Here's a very large semiprime: N = 299274023979912864896278377341791863851882963822276462493970736540519140853185037949526244111518584642464030275056341952320533303574841293319208222206628188165470634692153943037215768694676593099781134119555501118709660…

> Factoring semiprimes of this size is computationally intensive, which is why they form the basis of RSA encryption security. Not if it's divisible by 2. from sympy import isprime num = 29927402397991286489627837734179186385188296382227646249397073654051914085318503794952624411151858464246403027505634195232053330357484129331920822220662818816547063469215394303721576869467659309978113411955550111870966028627418736664…

Indeed that works for that case. But you can prompt yourselves, it will not always generate natural that are easy to validate with such shortcuts. So I don't think it invalidates the point I'm making.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#87

Post by shinycode » Sat, Aug 09, 2025, 9:01 AM UTC

gen220 wrote: post #85

shinycode wrote: post #8

I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?

Disclosure: my current employer has a product in this space (graphite.dev) IME the highest value (at the moment) is having an LLM integrated into the PR page, that reads your code + CI log, and effectively operates as a sanity check / semantic linter. A common workflow for us: is Draft PR -> Passes CI (inclusive of an LLM 'review') -> Published -> Passes Human review -> Scheduled to merge The goal is to get a higher…

Thank you for the feedback, it answers my question of the fact that as of now it’s just an other step in a human review. Nothing fully automatic (which is reassuring in a way) it’s just an other step to review & validate

Re: Benchmarking GPT-5 on 400 real-world code reviews

#88

Post by sigmoid10 » Sun, Aug 10, 2025, 11:47 PM UTC

qsort wrote: post #71

sigmoid10 wrote: post #66

Earlier quoted context omitted.

Ground truth evaluation is not that simple unless you are doing multiple-choice-style tests or something similar where the correctness of an answer can be determined by a simple process. Open ended natural language tasks like this one are incredibly difficult to evaluate and using LLMs as judge is not just the current standard, it is basically the only way to do it at scale economically.

The original comment was this: > So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that. The comment I replied to was: > That's how 99% of 'LLM benchmark numbers' circulating on the internet work. And that's just false. SWE-Bench verified isn't like this. Aider Polyglot isn't like this. SWE-Lancer Diamond isn't like this.…

Benchmarks that execute code are to some degree the only thing where you can automate testing at scale without humans in the loop, but even that has its caveats [1]. Regardless, when your output is natural language text (as is in this case), there is simply no viable alternative to measure accuracy economically. There is frankly no argument to be had here, because this is simply not achievable with current technology.

[1] https://openai.com/index/introducing-swe-bench-verified/