Without SWE-Bench though, how will AI models properly game their results to show ~5-10% gain each iteration?
Once a benchmark is known and there's billion of dollars on the line, obviously every company will game them.
SWE-bench Verified no longer measures frontier coding capabilities
101–110 of 209 posts
Re: SWE-bench Verified no longer measures frontier coding capabilities
#102Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…
I think the solution is a bunch of private trusted benchmarks, and averaging their announced results.
> averaging their announced results.
Obligatory XKCD: https://xkcd.com/937/
Re: SWE-bench Verified no longer measures frontier coding capabilities
#103I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…
> 93.6% (congrats Anthropic)
But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission"
0.191 * 0.594 > 1 - 0.936
Does this mean that the audited subset wasn't representative? Or that Anthropic is getting high answers through some shady means?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#104Benchmarks/evals are really hard and they become harder when there’s huge incentive to game them at an industry scale. ELT-Bench is another recent example. It was the first serious attempt at a benchmark for data engineering workloads, published about a year ago. A few days ago, a follow-up paper from a group that includes one of the original authors audited the benchmark itself. The team gfound that the benchmark ha…
It’s just hard to make them not part of the training data. We see this a bit with BrowseComp plus and other deep research datasets. Not because frontier labs are trying to cheat, but just from training on the full web. You need new datasets perpetually.
Or hidden benchmarks, though it's then harder to get people to trust the results.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#105Curiously Opus 4.7 claims to have a 87.6% pass rate and Mythos claims to have a 93.9% pass rate... leading to the conclusion that it's actually possible to "solve" the problems that OpenAI claims are incorrect.
If you read the mythos report, in which they discuss and account for contamination substantially, it still suggests that performance on SWE-bench verified is meaningful. Benchmarks, including SWE-bench can absolutely be gamed, but if you're not explicitly benchmaxxing, improving on SWE-bench still measures model improvements, at least up to the level of Mythos.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#106I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…
> 93.6% (congrats Anthropic) But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission" 0.191 * 0.594 > 1 - 0.936 Does this mean that the audited subset wasn't representative? Or that Anthropic is getting…
I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#107I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…
Both of them look pretty old?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#108I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…
Both of them look pretty old?
code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another
Re: SWE-bench Verified no longer measures frontier coding capabilities
#109[dead]
Re: SWE-bench Verified no longer measures frontier coding capabilities
#110Benchmarks/evals are really hard and they become harder when there’s huge incentive to game them at an industry scale. ELT-Bench is another recent example. It was the first serious attempt at a benchmark for data engineering workloads, published about a year ago. A few days ago, a follow-up paper from a group that includes one of the original authors audited the benchmark itself. The team gfound that the benchmark ha…
Would creating new benchmarks every month solve this problem?
Or create "blind" benchmarks.
10 groups of 3 researchers, all have their own benchmarks that they do not share (testing it without the authors knowing is a different problem, maybe they only run the benchmarks when the gen-pop has access to the models).
that's 10 different tests. Aggregate pass rates