Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

171–180 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#171

Post by seanmcdirmid »

stingraycharles wrote: post #164
seanmcdirmid wrote: post #122

Earlier quoted context omitted.

You can code your prompts to read and write an external world model on the side. This is what most people do who are seriously doing games with LLMs.

What do you mean with this? What is this world model, what does it capture?

You keep a document going called "state of the world", on every turn, you read this document in (as context), use it to help compute what happens, and based on what happens, create an updated "state of the world" document. You track important details so your LLM is consistent from turn to turn.

If you doing an RPG, which I guess is where this is more obvious, you track the play and enemy positions, their health, their moods and perhaps top thoughts, the state of important inanimate objects. if you break down the door, you update the door's state in the document. This is in contrast to just giving the LLM the previous turns and hoping it realizes the door is broken down later (just by statistical completion).

Re: SWE-bench Verified no longer measures frontier coding capabilities

#172

Post by gertlabs »

euleriancon wrote: post #170
orangebread wrote: post #53

Earlier quoted context omitted.

Wow. This benchmark definitely feels more accurate than the other rankings I've seen. My experience with gpt 5.4/5.5 is that they are technically flawless and if there are any technical issues that is because the input didn't provide enough clarity; that's not to say that it doesn't autonomously react to any issues during bug fixes or implementations, but it'll tend to nail its tasks without leaving behind gaps. Opus…

Are we looking at the same data? On that site I see that opus 4.7's and gpt 5.5's g scores are within each others confidence intervals, and both significantly ahead of the number 3 model. Your comment makes it sound like they are miles apart, which the benchmark doesn't seem to support. Edit: I looked at the data more and the two models are only basically equal when looking at the mean of all the tests. Gpt 5.5 signi…

Decision making refers to the environments where the LLM is called on every tick (like games with social communication), examples here: https://gertlabs.com/spectate.

Because GPT 5.5 just launched and those games take longer to accumulate data for, it just doesn't have enough samples yet. It will end up with a wider lead on Opus, I am sure. Coding evals always have large sample sizes on day 1. Good find, we should probably better adjust the weighting here for decision games with low match counts.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#173

Post by somenameforme »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

I'd add another thing here as well. Many take this sort of conspiratorial view like companies training on benchmarks would be some sort of underhanded intent at cheating. In reality, benchmarks also provide a way for companies to easily compare themselves to competitors and work to iteratively improve their own models, so there's a completely non-nefarious motivation to maximize scores on benchmarks.

In the end all it does is affirm what you're saying though. Benchmarks are essentially obsolete the moment they become recognized. I suppose it's just another iteration of Goodhart's Law.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#174

Post by fulafel »

energy123 wrote: post #103
ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

> 93.6% (congrats Anthropic) But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission" 0.191 * 0.594 > 1 - 0.936 Does this mean that the audited subset wasn't representative? Or that Anthropic is getting…

there's more details under the Too narrow and too wide tests heading.

It would be interesting to see a deeper investigation, into how the models are dealing with this and whether the successful ones seemed to be trained on the benchmark.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#175

Post by wtallis »

fibonacci112358 wrote: post #167
wtallis wrote: post #152

Earlier quoted context omitted.

That's a slightly different problem. There's no thing as saturation for a performance benchmark like SPEC; we can always conceive of a faster processor (even if we don't know how to build one). Saturation is the problem that once you are at (or near) 100% pass rate on a test of pass/fail questions, there's no room for the score to keep going up and the test has lost any power to discriminate between competing options…

Maybe OP was thinking about compilers "cracking" certain SPEC benchmarks: implementing exactly the optimization needed to boost a benchmark quite a lot, but that opt. probably won't apply to any other code out there (usually it's so targeted and risky with general C/C++ code that intentionally it doesn't work on anything else). That happened a couple of times over the years, I know about the Intel compiler cases for…

Intel's done it again recently, this time targeting Geekbench: https://www.intel.com/content/www/us/en/support/articles/000...

Both that and the SPEC compiler shenanigans are cheating by changing the test, not just over-specializing the product being benchmarked.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#176

Post by dannyw »

stingraycharles wrote: post #163
cjsaltlake wrote: post #106

Earlier quoted context omitted.

I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.

You can trust that a model that scores 40% vs a model that scores 90% is indeed worse. You can’t trust it that a model that scores 93% is better at software engineering than a model that scores 90%, because at that point it’s impossible to distinguish between recall and reasoning.

It’s honestly far better to just ignore SWEBench Verified in 2026. Multiple labs have noted issues with contamination, and achieving high scores require memorisation of what passes the prescriptive verifier; not what is a correct solution.

40% vs 90%? Sure.

70% vs 90%? _Absolutely meaningless_ as you are not measuring coding intelligence but “how well can the model cheat flaws in SWEBench Verified”, the former can certainly be better at coding even assuming no deliberate benchmaxxing / foul play.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#177

Post by mtrifonov »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

Still downstream of the actual issue. The benchmarks measure capability and the bottleneck stopped being capability a while ago.

What you actually want to measure on these models is what they can SEE in production. Context shape, retrieval quality, tool use, ability to compose state across turns. None of that is in SWE-bench because SWE-bench is shaped like a one-shot problem set and frontier coding work isn't shaped like that anymore.

Even a perfectly contamination-free benchmark would mostly test the wrong axis. The model is already at human-grad-student level on isolated problems. The leverage is in how it operates inside a larger system. And that's almost like, a taste/preference issue, and virtually impossible to objectively measure.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#178

Post by dannyw »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

Spend a hour or an afternoon creating your own eval harness with problems or workloads from your private repos or personal projects.

Use frontier LLMs to help create the harness and identify problems, but put in the effort to ensure your verifier is actually good and robust.

Then you have your own private benchmark, which makes new model releases a breeze instead of purely vibes or contaminated public benchmarks.

For extra props, add things you care about; such as reliability (eg deliberate noise injection, simple typo introduction in problems, variants, running each test multiple times).

At the end of the day however, the best LLM is the one you’re the most productive in. Frontier intelligence might be the main factor, but far from the only factor:

• How fast is it in the real world? How well does it understand your general style of prompting / guidance?

• How consistent and reliable is it? Does it exhibit laziness / hallucination of performing actions (and saying it does) it never performed?

• etc.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#179

Post by pkoiralap »

This was bound to happen either organically or inorganically. Make sure it performs well on the benchmarks. And it doesn't really matter if it doesn't generalize outside of it right? :D

Also similar: Graduate student descent. https://sciencedryad.wordpress.com/2014/01/25/grad-student-d...

Re: SWE-bench Verified no longer measures frontier coding capabilities

#180

Post by yalok »

gertlabs wrote: post #47

A better benchmark needs to be objectively scored, have multi-disciplinary, breadth, and be scalable (no single correct answer). That's what we designed at https://gertlabs.com . We put a lot of thought into it, and kept it mostly (not fully) related to problem solving through coding.

amazing to see Claude Code top models still way above all other models for C++ & Java, while GPT 5.5 is higher in Python & JS and others. Shows the skew in the training data sets, and maybe the go-to-market focus - with Anthropic focusing on enterprise customers much more than OpenAI?

Matches with my experience with Opus for C++.

C# results are empty - @gertlabs - any ETA for those?

Post reply on HN