11–20 of 121 posts
I am shocked— shocked —when a vendor cheats in order to increase their benchmark scores. I always tell my customers to ignore benchmarks and compare outcomes with their own workloads. Benchmarks are almost completely useless in the real world.
1. Did the benchmark authors not review the issues and make sure the solution was not present in the issue?
2. Are the issues locked after they’re included in the dataset? You’d think they would be immutable for reproducibility.
3. For the agents writing patches, is test running part of their inner loop validation? If they write a patch that makes the test pass, then the jobs done. Or is that validation step kept secret from the agent? I don’t see how unless the tests aren’t part of the repo.
So what we need is something like a versioned crowdsourced coding LLM eval dataset. Every quarter, you have a couple thousand volunteers provide 2 GitHub issues from the past 3 months, which are nontrivial to resolve, and where there exists strong test cases. Each volunteer then cross-checks 2 issues from other volunteers. The volunteers get 1 month free subscription to some AI service in return. This dataset is then…
Instead of resolving it, some leaders are further complicating their meaning
Such as OpenAI grading their benchmarks based on "how much money they made" or "how easy a model was convinced to hand over fake money".
I am shocked— shocked —when a vendor cheats in order to increase their benchmark scores. I always tell my customers to ignore benchmarks and compare outcomes with their own workloads. Benchmarks are almost completely useless in the real world.
> When we filtered out these problematic issues, the resolution rate of SWE-Agent+GPT-4 dropped from 12.47% to 3.97%. This matches my intuition about the coding performance of these models a lot better. I don't think any current coding benchmark accurately measures coding performance.
Earlier quoted context omitted.
Turns out "AI deep research reasoning agent" was just "we can print the training set"
...by piping it through the world's most inefficient echo function.
You need benchmarks with the following three properties: 1) No known solutions, so there's no "ground truth" dataset to train on 2) Presumably hard to solve 3) But easy to verify a solution if one is provided. This, of course, is easier done on the STEM side of things, but how do you automatically test creativity, or philosophical aptitude?
There’s a few things I’m not understanding here. 1. Did the benchmark authors not review the issues and make sure the solution was not present in the issue? 2. Are the issues locked after they’re included in the dataset? You’d think they would be immutable for reproducibility. 3. For the agents writing patches, is test running part of their inner loop validation? If they write a patch that makes the test pass, then t…