Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

151–160 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#151

Post by waterTanuki »

mxwsn wrote: post #96
civvv wrote: post #92

Earlier quoted context omitted.

This is likely true. I think model quality has stagnated and that its likely a non-trivial task to find a new improvement vector. Scaling the width of the model (which has been the driving force behind the speed of improvement thus far) seems to have reached its limit. It will be interesting to see the implications of this. Tooling can only do so much in the long term.

How do you know that width scaling has been the driving force of improvement?

I mean, it's not exactly a PhD level question. One can infer from the extreme demand of GPUs and DRAM + new data center construction that all the providers are banking on width.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#152

Post by wtallis »

kator wrote: post #134
ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

Those who fail to study history (or live through it) are doomed to repeat it. SPECint and SPECfp went through this exact movie: benchmark, saturate, retire, replace, repeat. The treadmill is the product. I don't have the solution just noticing the pattern.

That's a slightly different problem. There's no thing as saturation for a performance benchmark like SPEC; we can always conceive of a faster processor (even if we don't know how to build one). Saturation is the problem that once you are at (or near) 100% pass rate on a test of pass/fail questions, there's no room for the score to keep going up and the test has lost any power to discriminate between competing options.

However, both kinds of tests are susceptible to over-fitting: an LLM can be trained on the exact test questions, and a CPU can be designed with eg. branch predictors and cache sizes tuned specifically to handle a particular benchmark or workload.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#153

Post by davidheineman »

ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

SWE-bench is fantastic! IMO, the scrutiny is a byproduct of the adoption and success of the benchmark.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#154

Post by cwyers »

ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

[dead]

Re: SWE-bench Verified no longer measures frontier coding capabilities

#155

Post by red75prime »

cptskippy wrote: post #149
red75prime wrote: post #114

Earlier quoted context omitted.

Desarcasmed version: "I think that problems with Zork make those models virtually useless in programming tasks." Correct?

He said complicated code bases. LLMs are great at producing small snippets of code to address very targeted problems.

Great on small snippets of code, passable on larger pieces of code, great at finding vulnerabilities in large pieces of code, terrible in Zork. All-in-all, a jagged frontier that defies a simple sarcastic characterization.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#157

Post by MagicMoonlight »

cjsaltlake wrote: post #106
energy123 wrote: post #103

Earlier quoted context omitted.

> 93.6% (congrats Anthropic) But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission" 0.191 * 0.594 > 1 - 0.936 Does this mean that the audited subset wasn't representative? Or that Anthropic is getting…

I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.

This article says anthropic models can write out the entire benchmark solution set word for word from memory

Re: SWE-bench Verified no longer measures frontier coding capabilities

#158

Post by osti »

parentheses wrote: post #79

The timing makes me wonder if this is a direct response to Deepseek V4 having performance comparable to SOTA models.

This was published two months ago. Even though it was at a time that open source models are publishing comparable swe bench scores.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#159

Post by wredcoll »

z33k wrote: post #84
wredcoll wrote: post #82

This is somewhat tangential, but I want a model that can detect physical objects placed on top of a board from a picture/video, specifically warhammer 40k models. I want a model that can detect the actual units/models that are placed on top of the terrain/board so I can track how the models move during the game, but trying gemini and chatgpt they were absolutely rubbish.

Amiibo and Skylanders detect the pieces with NFC. Wiring up the whole board/ terrain with NFC readers would probably be difficult, though.

An interesting thought but at the moment I was just talking about analyzing a video lol

Re: SWE-bench Verified no longer measures frontier coding capabilities

#160

Post by anthonypasq96 »

jjmarr wrote: post #133
threepts wrote: post #52

Earlier quoted context omitted.

Optimal for judging actual reasoning ability rather than an LLM's ability to regurgitate knowledge from a necropost on HN/Reddit/Twitter from 2018.

I'm making an LLM agent that can play DS games. The biggest blocker is clicking on the right spot to move things around in space rather than reasoning abilities. Arc AGI seems to test that as well. Every game is a rectangular grid to make it as easy as possible yet the AIs still fail. I'm fairly certain the way forward isn't through agents directly interfacing with UIs but through agents using scripts and other tools…

the whole point of arc-agi 3 is that if models are AGI then they should be able to solve the same tasks as humans do given the same information, but they cant. allowing scripts and harnesses and whatnot completely defeats the purpose.
Post reply on HN