Benchmarking GPT-5 on 400 real-world code reviews
1–10 of 88 posts
Re: Benchmarking GPT-5 on 400 real-world code reviews
#2Can you benchmark Kimi K2 and GLM 4.5 as well? Would be interesting to see where they land.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#3> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization.
This is key.
Public benchmarks are essentially trust-based and the trust just isn't there.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#4> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score.
So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#5> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization. This is key. Public benchmarks are essentially trust-based and the trust just isn't there.
Unless you're running the LLM yourself (locally), private benchmarks are also trust-based, aren't they?
Re: Benchmarking GPT-5 on 400 real-world code reviews
#6> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization. This is key. Public benchmarks are essentially trust-based and the trust just isn't there.
How does this ensure models haven’t seen it during training - is it a different benchmark per model release?
Re: Benchmarking GPT-5 on 400 real-world code reviews
#7> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
Yes, especially as models are known to have a preference towards outputs of models in the same family. I suspect this leaderboard would change dramatically with different models as the judge.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#8I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?
Re: Benchmarking GPT-5 on 400 real-world code reviews
#9> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization. This is key. Public benchmarks are essentially trust-based and the trust just isn't there.
Unless you're running the LLM yourself (locally), private benchmarks are also trust-based, aren't they?
Yes, but in a case like this it's a neutral third-party running the benchmark. So there isn't a direct incentive for them to favor one lab over another.
With public benchmarks we're trusting the labs not to cheat. And it's easy to "cheat" accidentally - they actually need to make a serious effort to not contaminate the training data.
And there's massive incentives for the labs to cheat in order to get the hype going around their launch and justify their massive investments in training. It doesn't have to be the CEO who's directing it. Can even be one/a few researchers who are responsible for a specific area of model performance and are under tremendous pressure to deliver.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#10> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
Yes, especially as models are known to have a preference towards outputs of models in the same family. I suspect this leaderboard would change dramatically with different models as the judge.
They are different models already but yes, I already let ChatGPT judge Claude's work for the same reason.