Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Benchmarking GPT-5 on 400 real-world code reviews

qodo.ai

Post reply on HN
Print view View original

31–40 of 88 posts

Re: Benchmarking GPT-5 on 400 real-world code reviews

#31

Post by rullelito »

8-prime wrote: post #14

Asking GPT 4o seems like an odd choice. I know this is not quite comparable to what they were doing, but asking different LLMs the following question > answer only with the name nothing more norting less.what currently available LLM do you think is the best? Resulted in the following answers: - Gemini 2.5 flash: Gemini 2.5 Flash - Claude Sonnet 4: Claude Sonnet 4 - Chat GPT: GPT-5 To me its conceivable that GPT 4o wo…

Without knowing too much about ML training, generated output from the own model must be much easier to understand since it generates data that is more likely to be similar to the training set? Is this correct?

Re: Benchmarking GPT-5 on 400 real-world code reviews

#32

Post by qsort »

raincole wrote: post #13
comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

That's how 99% of 'LLM benchmark numbers' circulating on the internet work.

No, they aren't. Most benchmarks use ground truth, not evaluation by another LLM. Using another LLM as verifier, aside from the obvious "quis custodiet custodes ipsos", opens an entire can of worms, such as the fact that there could be systematic biases in the evaluation. This is not in and of itself disqualifying but it should be addressed, and the article doesn't even say anything.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#33

Post by shikon7 »

comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

Also, using an OpenAI model to judge the performance of an OpenAI model seems prone to all kinds of biases.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#34

Post by Leherenn »

shinycode wrote: post #8

I’m curious to know how people use PR review platforms with LLMs. Because what I feel is that I need to do the review and then review the review of the LLM which is more work in the end. If I don’t review anymore (or if no one does it) knowledge is kind of lost. It surely depends on team size but do people use those to only to have better hints or to accelerate reviews with no/low overlook ?

Only has a sanity check/better hints. But I use it for my own PRs, not others'. Usually it's not much to review and easy to agree/disagree with.

I haven't found it to be really useful so far, but it's also very little added work, so for now I keep on using it. If it saves my ass even just once, it will probably be worth it overall.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#35

Post by monkeydust »

8-prime wrote: post #14

Asking GPT 4o seems like an odd choice. I know this is not quite comparable to what they were doing, but asking different LLMs the following question > answer only with the name nothing more norting less.what currently available LLM do you think is the best? Resulted in the following answers: - Gemini 2.5 flash: Gemini 2.5 Flash - Claude Sonnet 4: Claude Sonnet 4 - Chat GPT: GPT-5 To me its conceivable that GPT 4o wo…

I know from our research models do exhibit bias when used this way as llm as a judge...best to use a totally different foundation company for the judge.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#36

Post by sensanaty »

with wrote: post #15
comex wrote: post #4

> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.

It’s a widely accepted eval technique and it’s called “llm as a judge”

Accepted by whom, the people shoving AI down our throats?

Re: Benchmarking GPT-5 on 400 real-world code reviews

#37

Post by jondwillis »

rullelito wrote: post #31
8-prime wrote: post #14

Asking GPT 4o seems like an odd choice. I know this is not quite comparable to what they were doing, but asking different LLMs the following question > answer only with the name nothing more norting less.what currently available LLM do you think is the best? Resulted in the following answers: - Gemini 2.5 flash: Gemini 2.5 Flash - Claude Sonnet 4: Claude Sonnet 4 - Chat GPT: GPT-5 To me its conceivable that GPT 4o wo…

Without knowing too much about ML training, generated output from the own model must be much easier to understand since it generates data that is more likely to be similar to the training set? Is this correct?

I don’t think so. The training data, or some other filter applied to the output tokens, is resulting in each model indicating that it is the best.

The self-preference is almost certainly coming from post-processing, or more likely because the model name is inserted into the system prompt.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#38

Post by ben_w »

jacquesm wrote: post #28
with wrote: post #15

Earlier quoted context omitted.

It’s a widely accepted eval technique and it’s called “llm as a judge”

Accepted does not mean correct. It's like using a rubber yardstick as the means to figure out who won the pumpkin growing competition.

I'd say it's worse than that, a rubber ruler still has a definite length when not under tension etc.

This might be more like asking amateur painters to each paint a picture of a different one of the pumpkins, then judging each other's paintings without seeing the actual pumpkin that painting was based on.

Re: Benchmarking GPT-5 on 400 real-world code reviews

#39

Post by jondwillis »

Idea: randomized next token prediction passed to a bunch of different models on a rotating basis.

It’d be harder to juice benchmarks if a random sample of ~100 top models were randomly sampled in this manner for output tokens while evaluating the target model’s output.

On second thought, I’m slapping AGPL on this idea. Please hire me and give me one single family house in a California metro as a bonus. Thanks.

Post reply on HN