Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

201–209 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#201

Post by CamperBob2 »

mnky9800n wrote: post #197
CamperBob2 wrote: post #77

Earlier quoted context omitted.

Actually the Zorks weren't deterministic, especially Zork II. The Wizard could F you over pretty badly if he appeared at an inopportune time.

I feel like you are being pedantic. There are very few parts of Zork that are not static to the game. Yes the thief shows up randomly but that’s not the main point of the game.

It is not the least bit pedantic. Games were meaner back then. If you're on a time (turn)-limited section of the game, or in a vulnerable spot like the volcano, random encounters with the wizard could render the game unwinnable without dying, which would completely wreck a benchmark. Same for the thief in Zork 1. If he randomly steals your light source, you're done for. Or if the RNG dictates that you lose the fight with the troll.

Can't recall anything like that in Zork 3. (Edit: apparently you could get shot randomly when using the time machine in the Royal Museum.)

Re: SWE-bench Verified no longer measures frontier coding capabilities

#202

Post by mrothroc »

ofirpress wrote: post #99

I'm a co-creator of SWE-bench: 1. SWE-bench Verified is now saturated at 93.9% (congrats Anthropic), but anyone who hasn't reached that number yet still has more room for growth. 2. SWE-bench Multilingual and SWE-bench Multimodal (which we'll open source in the next month) are still unsatured. 3. All benchmarks and benchmark paradigms eventually become saturated. That's why the SWE-bench team has worked hard on build…

From a verification-topology angle, what makes algotune.io contamination-resistant? Is it because the correctness oracle is a performance metric (which can't be memorized) rather than a fixed test that can?

Re: SWE-bench Verified no longer measures frontier coding capabilities

#204

Post by anthonypasq »

knollimar wrote: post #55
threepts wrote: post #52

Earlier quoted context omitted.

Optimal for judging actual reasoning ability rather than an LLM's ability to regurgitate knowledge from a necropost on HN/Reddit/Twitter from 2018.

a small harness that stores text files and manages context could be useful, otherwise you lose all ability to measure that skill (and that's important because it represents real world use cases on large code bases)

arc agi isnt testing a models ability to store files and code things. its testings its ability to reason through puzzles given the same information as a human

Re: SWE-bench Verified no longer measures frontier coding capabilities

#205

Post by 0error-ob »

stingraycharles wrote: post #163
cjsaltlake wrote: post #106

Earlier quoted context omitted.

I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.

You can trust that a model that scores 40% vs a model that scores 90% is indeed worse. You can’t trust it that a model that scores 93% is better at software engineering than a model that scores 90%, because at that point it’s impossible to distinguish between recall and reasoning.

[dead]

Re: SWE-bench Verified no longer measures frontier coding capabilities

#206

Post by knollimar »

anthonypasq wrote: post #204
knollimar wrote: post #55

Earlier quoted context omitted.

a small harness that stores text files and manages context could be useful, otherwise you lose all ability to measure that skill (and that's important because it represents real world use cases on large code bases)

arc agi isnt testing a models ability to store files and code things. its testings its ability to reason through puzzles given the same information as a human

if you tested my ability to reason and you gave me some challenging problems that involved arithmetic, it might be a better test if you gave me a scratch pad so I don't mess up the reasoning parts by failing arithmetic.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#208

Post by falcor84 »

anthonypasq wrote: post #204
knollimar wrote: post #55

Earlier quoted context omitted.

a small harness that stores text files and manages context could be useful, otherwise you lose all ability to measure that skill (and that's important because it represents real world use cases on large code bases)

arc agi isnt testing a models ability to store files and code things. its testings its ability to reason through puzzles given the same information as a human

But that's the thing, as a human faced with a problem I'd often say "Sure, just let me get a pen, some paper and a calculator". Why shouldn't we make it easy for AIs to use their tools of choice?
Post reply on HN