Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

131–140 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#131

Post by [deleted] »

civvv wrote: post #92
kqr wrote: post #50

It was never that great, it seems. For all of 2025 there was virtually no improvement in the rate at which models produced quality code. They only got better at passing automated tests. https://entropicthoughts.com/no-swe-bench-improvement

This is likely true. I think model quality has stagnated and that its likely a non-trivial task to find a new improvement vector. Scaling the width of the model (which has been the driving force behind the speed of improvement thus far) seems to have reached its limit. It will be interesting to see the implications of this. Tooling can only do so much in the long term.

[deleted]

Re: SWE-bench Verified no longer measures frontier coding capabilities

#132

Post by tedsanders »

DeathArrow wrote: post #59

So Opus 4.7 and Mythos are solving problems that are impossible to solve?

Whether a problem is "good" or "bad" is not always objective or simple.

For example, you can have problems that are underspecified, with hardcoded tests for a particular solution (out of multiple possible solutions). If your solution works fine but used a different function name than the one hardcoded in the tests, you can unfairly score 0.

When an eval has underspecified problems like these, you can still score 100% if you remember the original solution from your training data or if you just have taste similar to the original human authors. And both of these qualities - good memory and good taste - are great, but they'll be rewarded unfairly relative to a model that still did exactly what it was asked but in a different way than the hardcoded tests expected.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#133

Post by jjmarr »

threepts wrote: post #52
falcor84 wrote: post #36

Earlier quoted context omitted.

They focus on minimizing the number of moves and don't allow any harness whatsoever, putting the bar extremely high. The current top verified contender (Claude Opus 4.6) is at only 0.45%. But with how new it is, I expect a lot of improvement in the next generation of models.

Optimal for judging actual reasoning ability rather than an LLM's ability to regurgitate knowledge from a necropost on HN/Reddit/Twitter from 2018.

I'm making an LLM agent that can play DS games. The biggest blocker is clicking on the right spot to move things around in space rather than reasoning abilities.

Arc AGI seems to test that as well. Every game is a rectangular grid to make it as easy as possible yet the AIs still fail.

I'm fairly certain the way forward isn't through agents directly interfacing with UIs but through agents using scripts and other tools to interact with the interface. That's why harnesses are so critical to performance on tasks like this.

I would like a version of Arc AGI that tests the agent's ability to dynamically create these harnesses.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#134

Post by kator »

ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

Those who fail to study history (or live through it) are doomed to repeat it.

SPECint and SPECfp went through this exact movie: benchmark, saturate, retire, replace, repeat. The treadmill is the product.

I don't have the solution just noticing the pattern.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#135

Post by kator »

cjsaltlake wrote: post #106
energy123 wrote: post #103

Earlier quoted context omitted.

> 93.6% (congrats Anthropic) But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission" 0.191 * 0.594 > 1 - 0.936 Does this mean that the audited subset wasn't representative? Or that Anthropic is getting…

I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.

> models that aren't over-optimized for it.

But how do you know the model was over-optimized for it or just really good?

Re: SWE-bench Verified no longer measures frontier coding capabilities

#136

Post by trueno »

cbg0 wrote: post #75
Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

> let the community decide Which community are we talking about? The professionals with 10+ years experience using LLMs, the vibe coders that have no experience writing code and everyone in between? If you read some of the online communities the experiences with the models all over the place, some compare GPT 5.5 to the second coming of JC while others think it's stupider than 5.4. I personally don't have time to bui…

yea lol i think the community on this one is woefully unqualified to call any shots here. the goalposts are basically teleporting and everyone's aligning success with their own incredibly vague, personally created agentic non-deterministic workflows success. there's like no real answers coming from "the community" in this space at the moment, it's vividly similar to cryptocurrency cycles. most importantly, like you say, vibe coders are going to be the largest subset of the community and probably the most unqualified to assess performance because they're mostly clueless to how things work under the hood.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#137

Post by wtallis »

embedding-shape wrote: post #19
vintagedave wrote: post #11

> We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions, despite our best efforts in improving on this in the initial creation of SWE-bench Verified. Is this saying a quarter* of the questions and answers were wrong, this whole time?! If so, how was this ever, in any way, a va…

> Is this saying a quarter of the questions and answers were wrong, this whole time?! No, they're saying 59.4% of the 27.6% subset had flawed test cases I think. > If so, how was this ever, in any way, a valid measurement? Benchmarks essentially aren't, for practical concerns anyways. They don't represent your use case, and they don't represent any and all use cases, they're valid for measuring exactly what's include…

I'm not sure people are really trying to interpret this kind of benchmark as being accurate in gauging the magnitude of improvement. It seems pretty obvious that doubling your score on some benchmark where 100% means "correctly answered all of these specific problems" doesn't translate directly to performing twice as well on all problems. I think what people want from these benchmarks—and what they do get to some extent—is answering the question of "is model A better than model B", especially the subset of "is this local model better than last year's frontier online model".

The marketing departments touting each model do want to claim superiority on the basis of slivers of percentage points, and that's probably always a stronger claim than the test results can reasonably support. And the benchmarks are obviously susceptible to cheating and overfitting. But when the scores aren't saturated and do show a big discrepancy, that kind of result usually seems to align with what people report from actually trying to use the models in the relevant problem space.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#138

Post by dominotw »

ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

how hard is it create one of these for my company that models most of the work we do at my company.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#139

Post by jddj »

For the most part I think we get the benchmarks we deserve.

Many SWE-bench passing PRs would not be merged: https://news.ycombinator.com/item?id=47341645

Top model SWE bench scores may be skewed by git history leaks: https://news.ycombinator.com/item?id=45214670

Re: SWE-bench Verified no longer measures frontier coding capabilities

#140

Post by varispeed »

gruez wrote: post #57
varispeed wrote: post #22

Issue with these benchmark also is that they measure a model you are unlikely going to be routed to. My experience with Anthropic is that despite using Opus 4.6 and 4.7, most of the time the performance is matching low B parameter Qwen. I think there should be a way to verify what model is actually being used to process prompts - that should be independently verified. At the moment it is so bad, you have to ask verif…

> My experience with Anthropic is that despite using Opus 4.6 and 4.7, most of the time the performance is matching low B parameter Qwen. Is this just the next level of the "they're serving quantized models!" theory?

Not a theory buy lived experience. You never know when you get the nerfed session.
Post reply on HN