Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-5 on 400 real-world code reviews

qodo.ai

Post reply on HN
Print view View original

21–30 of 88 posts

Re: Benchmarking GPT-5 on 400 real-world code reviews

#21

Post by stpedgwdgfhgdd »

shinycode wrote: post #8

I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?

I give the MR id to CC and let it review. I have glab cli installed so it knows how to pull and even add a comment. Unfortunately not at all specific line number afaict. I also have Atlassian MCP, so CC can also add a comment in the Jira work item (fka issue).

Re: Benchmarking GPT-5 on 400 real-world code reviews

#23

Post by vohk »

timbilt wrote: post #9
laggyluke wrote: post #5

Earlier quoted context omitted.

Unless you're running the LLM yourself (locally), private benchmarks are also trust-based, aren't they?

Yes, but in a case like this it's a neutral third-party running the benchmark. So there isn't a direct incentive for them to favor one lab over another. With public benchmarks we're trusting the labs not to cheat. And it's easy to "cheat" accidentally - they actually need to make a serious effort to not contaminate the training data. And there's massive incentives for the labs to cheat in order to get the hype going…

The problem is when using a model hosted by those labs (ex: OpenAI only allowed access to o3 through their own direct API, not even Azure), there still exists a significant risk of cheating.

There's a long history of that sort of behaviour. ISPs gaming bandwidth tests when they detect one is being run. Software recognizing being run in a VM or on a particular configuration. I don't think it's a stretch to assume some of the money at OpenAI and others has gone into spotting likely benchmark queries and throwing on a little more compute or tagging them for future training.

I would be outright shocked if most of these benchmarks are even attempting serious countermeasures.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#24

Post by jacquesm »

ImageXav wrote: post #7
comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

Yes, especially as models are known to have a preference towards outputs of models in the same family. I suspect this leaderboard would change dramatically with different models as the judge.

I don't care about either method. The ground truth should be what a human would do, not what a model does.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#25

Post by Lionga »

carlob wrote: post #20
Lionga wrote: post #16

Company selling AI Reviews says AI Reviews great! In other news water is wet.

Company selling AI Reviews says its AI Review of AI Reviews concluded AI reviews are great! In other news water is wet (as assessed by more water). FTFY

My AI Review says your comment is 100% perfect (this comment was written by ChatGPT 5)

Re: Benchmarking GPT-5 on 400 real-world code reviews

#26

Post by magicalhippo »

with wrote: post #15
comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

It’s a widely accepted eval technique and it’s called “llm as a judge”

Shouldn't one review the ratings of say a random 1% to ensure it's performing as expected?

Re: Benchmarking GPT-5 on 400 real-world code reviews

#28

Post by jacquesm »

with wrote: post #15
comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

It’s a widely accepted eval technique and it’s called “llm as a judge”

Accepted does not mean correct. It's like using a rubber yardstick as the means to figure out who won the pumpkin growing competition.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#29

Post by jacquesm »

timbilt wrote: post #3

> Unlike many public benchmarks, the PR Benchmark is private, and its data is not publicly released. This ensures models haven’t seen it during training, making results fairer and more indicative of real-world generalization. This is key. Public benchmarks are essentially trust-based and the trust just isn't there.

Then you just need to use different data the next time you evaluate. That is much more indicative of real-world generalization: after all, you don't normally do multiple PRs on the same pieces of code. The current approach risks leaking the dataset selectively and/or fudging the results because they can't be verified. Transparency is key when doing this kind of benchmark, so now we have to trust the entity doing the benchmarking rather than independent verification of the results and with the amount of money that is at stake here I don't think that's the way to go.
Post reply on HN