Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

111–120 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#111

Post by cjsaltlake »

kqr wrote: post #50

It was never that great, it seems. For all of 2025 there was virtually no improvement in the rate at which models produced quality code. They only got better at passing automated tests. https://entropicthoughts.com/no-swe-bench-improvement

But, that's an enormous source of coding productivity, and it's why Anthropic is worth billions... The reason SWE-bench has been so successful and useful for coding is that software engineering has a ton of tradition and infrastructure for making and using automated tests.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#112

Post by cjsaltlake »

rustyhancock wrote: post #61

I think an Olympiad format is better. But the financial incentive is such that it might be near impossible to stop leaks. I.e. A panel comes up with a series of problems. Like advent of code or project Euler but more complex and constricted. Benchmark outcomes could be performance points and measure of cost, time to solution (well token count really). A couple times per year it's run. It avoids overfitting. Overtime…

SWE-bench was created to replace olympiad coding benchmarks. I think past olympiad coding benchmarks were much worse representative of real-world coding than something like SWE-bench, which is derived from real units of labor.

Further, olympiad style benchmarks are arguably easier to contaminate / memorize unless you refresh it regularly; but that goes for SWE-bench too.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#113

Post by hashmap »

1a527dd5 wrote: post #3

This feels very much like "we are now moving the goal posts".

It does, and it should. With each iteration getting closer to the goalposts exposes the flaws in the goalposts, and then you try to make better goalposts. The problem people seem to have with the goalposts moving is they assume the goalpost makers either made good goalposts or thought they made good goalposts, but the actual process is "do the best we can at the moment and update when we get better information".

Re: SWE-bench Verified no longer measures frontier coding capabilities

#114

Post by red75prime »

malfist wrote: post #88
kqr wrote: post #40

Earlier quoted context omitted.

I have worked on similar problems. See e.g. [1]. The LLMs I have tested have terrible world models and intuitions for how actions change the environment. They're also not great at discerning and pursuing the right goals. They're like an infinitely patient five-year old with amazing vocabulary. [1]: https://entropicthoughts.com/updated-llm-benchmark (more descriptions available in earlier evaluations referenced from t…

I'm going to ignore all that and tell my developers working in complicated codebases that they have to use AI. I'm sure comprehending side effects in a world building text adventure is completely different that understanding spaghetti code

Desarcasmed version: "I think that problems with Zork make those models virtually useless in programming tasks." Correct?

Re: SWE-bench Verified no longer measures frontier coding capabilities

#115

Post by Bombthecat »

cjsaltlake wrote: post #108
Bombthecat wrote: post #107

Earlier quoted context omitted.

Both of them look pretty old?

code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another

I mean the data / benchmarks

Re: SWE-bench Verified no longer measures frontier coding capabilities

#116

Post by cpard »

softwaredoug wrote: post #93
cpard wrote: post #81

Benchmarks/evals are really hard and they become harder when there’s huge incentive to game them at an industry scale. ELT-Bench is another recent example. It was the first serious attempt at a benchmark for data engineering workloads, published about a year ago. A few days ago, a follow-up paper from a group that includes one of the original authors audited the benchmark itself. The team gfound that the benchmark ha…

It’s just hard to make them not part of the training data. We see this a bit with BrowseComp plus and other deep research datasets. Not because frontier labs are trying to cheat, but just from training on the full web. You need new datasets perpetually.

That’s true. it also depends heavily on the type of task, not everything is equally represented on the web today and it remains to be seen if this is going to change or not.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#117

Post by cpard »

stavros wrote: post #104
softwaredoug wrote: post #93

Earlier quoted context omitted.

It’s just hard to make them not part of the training data. We see this a bit with BrowseComp plus and other deep research datasets. Not because frontier labs are trying to cheat, but just from training on the full web. You need new datasets perpetually.

Or hidden benchmarks, though it's then harder to get people to trust the results.

The trust issue might be solved by having standardisation bodies created, similar to W3C or even TPC, although TPC didn’t end that well.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#118

Post by gertlabs »

cjsaltlake wrote: post #108
Bombthecat wrote: post #107

Earlier quoted context omitted.

Both of them look pretty old?

code clash I think would be quite hard to game or contaminate unintentionally; considering that models need to compete against one another

https://gertlabs.com already does this at scale.

An industry-standard benchmark shouldn't be hosted or designed by a lab producing the models, regardless.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#119

Post by marlburrow »

The "private benchmarks" suggestion comes up every time, but I think there's a more interesting axis: benchmarks built on top of already-public, already-stable test instruments. SWE-bench is fundamentally a corpus that lives on GitHub — once it ships, it leaks into training data automatically. Benchmarks built on contested qualitative instruments (psych tests, opinion surveys) have a different contamination profile because the correct answer doesn't exist in the training corpus to memorize — only the question does.

That doesn't help for measuring coding ability specifically (you fundamentally need a code-correctness oracle), but for capability axes where the "answer" is a stated position rather than a verifiable fact, public + stable can still be useful. The SWE-bench problem isn't really "public", it's "public + has a fixed correct answer".

Re: SWE-bench Verified no longer measures frontier coding capabilities

#120

Post by wavemode »

embedding-shape wrote: post #19
vintagedave wrote: post #11

> We audited a 27.6% subset of the dataset that models often failed to solve and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submissions, despite our best efforts in improving on this in the initial creation of SWE-bench Verified. Is this saying a quarter* of the questions and answers were wrong, this whole time?! If so, how was this ever, in any way, a va…

> Is this saying a quarter of the questions and answers were wrong, this whole time?! No, they're saying 59.4% of the 27.6% subset had flawed test cases I think. > If so, how was this ever, in any way, a valid measurement? Benchmarks essentially aren't, for practical concerns anyways. They don't represent your use case, and they don't represent any and all use cases, they're valid for measuring exactly what's include…

> No, they're saying 59.4% of the 27.6% subset had flawed test cases I think.

That being said, they didn't audit the other 72.4%, right? So it's likely that there are way more flawed problems throughout the full set?

Post reply on HN