Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

41–50 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#41

Post by WarmWash »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

An easy way to make coding benchmarks viable again is to initialize the models with 200k of distracting or unrelated tokens in their context. Or even just run the tests sequentially in the same context and see how far the model gets before it unwinds.

These benchmarks are always greenfield, but people want a model that can deal with a rotted context.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#42

Post by MattRix »

gpm wrote: post #25

Curiously Opus 4.7 claims to have a 87.6% pass rate and Mythos claims to have a 93.9% pass rate... leading to the conclusion that it's actually possible to "solve" the problems that OpenAI claims are incorrect.

The problem isn’t that the tasks are impossible to solve, it’s that they’re underspecified and/or impossible to solve consistently (ex. because a test is expecting the solution function to have a specific name that wasn’t specified in the task itself).

So maybe Anthropic runs Mythos through the benchmark 10000 times and takes the highest score, who knows?

Re: SWE-bench Verified no longer measures frontier coding capabilities

#43

Post by MattRix »

iLoveOncall wrote: post #10
neversupervised wrote: post #4

Earlier quoted context omitted.

But this is the good kind of goalpost moving

Only if you didn't read the article. They're saying they need to move on from it because the benchmark is flawed (without bringing in proof) and that's why they can't hit 100%. It's not a "our models are so good that the benchmark is too easy" thing.

How can you say “without bringing in proof” when there is literally proof in the article?

Re: SWE-bench Verified no longer measures frontier coding capabilities

#44

Post by cyanydeez »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

a good benchmark would probably porting a selected repo to another language. then clear context notes, and have it port it back.

as long as theres a test framework, you could gauge success deterministically.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#46

Post by WarmWash »

mnky9800n wrote: post #29
Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench

The open models only give the SOTA models a run for their money on gameable benchmarks. On the semi-private ARC-AGI 2 sets they do absolutely awfully (It might be too expensive, but I would be interested in the benchmarks for the current crop of SOTA models.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#47

Post by gertlabs »

A better benchmark needs to be objectively scored, have multi-disciplinary, breadth, and be scalable (no single correct answer).

That's what we designed at https://gertlabs.com. We put a lot of thought into it, and kept it mostly (not fully) related to problem solving through coding.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#49

Post by gpm »

MattRix wrote: post #42
gpm wrote: post #25

Curiously Opus 4.7 claims to have a 87.6% pass rate and Mythos claims to have a 93.9% pass rate... leading to the conclusion that it's actually possible to "solve" the problems that OpenAI claims are incorrect.

The problem isn’t that the tasks are impossible to solve, it’s that they’re underspecified and/or impossible to solve consistently (ex. because a test is expecting the solution function to have a specific name that wasn’t specified in the task itself). So maybe Anthropic runs Mythos through the benchmark 10000 times and takes the highest score, who knows?

We actually know that a "100% pass rate" is trivially possible: https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/

Anthropic p-hacking the benchmark strikes me as cheating, and somewhat unlikely. Mythos figuring out how to cheat at the benchmark strikes me as much more likely.

But if that hypothesis is the explanation the interesting part is Opus 4.7 (but not 4.6) seems to be doing the same.

Post reply on HN