How can o4-mini be at 57, and sonnet-4 is at 39? This is way off, o4-mini is not even in the top 5 of coding agents.
Benchmarking GPT-5 on 400 real-world code reviews
61–70 of 88 posts
Re: Benchmarking GPT-5 on 400 real-world code reviews
#62Earlier quoted context omitted.
Only has a sanity check/better hints. But I use it for my own PRs, not others'. Usually it's not much to review and easy to agree/disagree with. I haven't found it to be really useful so far, but it's also very little added work, so for now I keep on using it. If it saves my ass even just once, it will probably be worth it overall.
> If it saves my ass even just once, it will probably be worth it overall. That's a common fallacy of safety by the way :) It could very well "save your ass" just once (whatever that means) while costing you more in time, opportunity, effort, or even false sense of safety, to generate more harm than it will ultimately save you.
Sure, but so far the cost is very minimal. Like 1 minute per PR on average. A crash in production and the subsequent falloffs is probably a good week of work and quite a bit of stress. That gives me quite a few PRs.
And it's not even safety critical code.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#63Earlier quoted context omitted.
Am I missing something? If LLM-1 is supposed to judge LLM-2, doesn't LLM-1 have to be better than LLM-2? If LLM-1 is only 40% as good at coding as LLM-2, why would you trust the LLM with the lesser knowledge?
At the heart of the P vs NP problem lies the observation that solution verification seems to be much easier than solution generation. If that applies in this context is another question but I think it is not unreasonable to assume that the judge needs to be less powerful than the performer. Or in other words, I don't need to be a chef myself to decide if a meal is good or not.
It's usually easier to create a false statement than to check whether it's false.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#64Earlier quoted context omitted.
At the heart of the P vs NP problem lies the observation that solution verification seems to be much easier than solution generation. If that applies in this context is another question but I think it is not unreasonable to assume that the judge needs to be less powerful than the performer. Or in other words, I don't need to be a chef myself to decide if a meal is good or not.
That really doesn't hold for all problems. You can imagine any number of problems where a valid solution is easier, complexity wise, to generate than it is to validate. A trivial example is semiprime factorization. Easy to generate any semiprime, hard to factor.
Sure, it was never my intention to make it seem like a general statement, just highlighting that there is a large class of problems for which it is true.
As you point out there are many problems that higher complexity classes than NP.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#65> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
It undermines the private benchmark approach if the evaluation is done that way.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#66Earlier quoted context omitted.
That's how 99% of 'LLM benchmark numbers' circulating on the internet work.
No, they aren't. Most benchmarks use ground truth, not evaluation by another LLM. Using another LLM as verifier, aside from the obvious "quis custodiet custodes ipsos", opens an entire can of worms, such as the fact that there could be systematic biases in the evaluation. This is not in and of itself disqualifying but it should be addressed, and the article doesn't even say anything.
Ground truth evaluation is not that simple unless you are doing multiple-choice-style tests or something similar where the correctness of an answer can be determined by a simple process. Open ended natural language tasks like this one are incredibly difficult to evaluate and using LLMs as judge is not just the current standard, it is basically the only way to do it at scale economically.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#67> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
> Hard to tell what to make of that.
It's not hard. You are visiting a website with an .ai domain. You already know what the conclusions will be.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#68Earlier quoted context omitted.
Also, using an OpenAI model to judge the performance of an OpenAI model seems prone to all kinds of biases.
Am I missing something? If LLM-1 is supposed to judge LLM-2, doesn't LLM-1 have to be better than LLM-2? If LLM-1 is only 40% as good at coding as LLM-2, why would you trust the LLM with the lesser knowledge?
At least use something like Zen MCP’s Consensus tool to gain a consensus around a large variety of models.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#69> Each model’s responses are ranked by a high-performing judge model — typically OpenAI’s o3 — which compares outputs for quality, relevance, and clarity. These rankings are then aggregated to produce a performance score. So there's no ground truth; they're just benchmarking how impressive an LLM's code review sounds to a different LLM. Hard to tell what to make of that.
It's almost too on the nose to be satire, yet here we are.
Re: Benchmarking GPT-5 on 400 real-world code reviews
#70Earlier quoted context omitted.
Also, using an OpenAI model to judge the performance of an OpenAI model seems prone to all kinds of biases.
Am I missing something? If LLM-1 is supposed to judge LLM-2, doesn't LLM-1 have to be better than LLM-2? If LLM-1 is only 40% as good at coding as LLM-2, why would you trust the LLM with the lesser knowledge?
[deleted]