Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Separating signal from noise in coding evaluations

openai.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Separating signal from noise in coding evaluations

#31

Post by janalsncm »

Based on the numbers here it seems there’s less than 800 tasks in the entire benchmark. That is enough for a handful of engineers to comb through in a week (which is what OpenAI eventually did here).

On the one hand, kudos to them for actually doing that work.

On the other hand, garbage in, garbage out. It’s a bit embarrassing for the original authors to have not actually checked, and it’s embarrassing for everyone downstream to have not checked either.

Also if you check the article, although an LLM did find issues, it tended to underestimate issues that professional software engineers found.

Re: Separating signal from noise in coding evaluations

#32

Post by jheitmann »

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a pretty solid path to take.

Re: Separating signal from noise in coding evaluations

#34

Post by ACCount37 »

bellowsgulch wrote: post #4

Seems like depending on your field these days, the hot thing to do is build your own private benchmarks. In my own testing, no frontier model knows how to replicate an original 1990s Super Soaker prototype design, which for the most part, should be almost completely possible with Home Depot parts. They just don't understand PVC parts, triggers, etc.

It's a "commonsense spatial reasoning/problem solving" kind of problem. LLMs fail at spatial reasoning forever.

What humans "easily" solve in seconds with raw spatial reasoning LLMs often find easier to solve by invoking A* or a constraint solver.

Might be that text data is particularly bad at teaching that to LLMs. Or that being good at spatial reasoning requires true recurrence, and autoregressive chain of thought is a poor substitute. Or it might be that human brain was optimized by evolution for solving spatial problems in open ended 3D environments for hundreds of millions of years, optimized for language for mere hundreds of thousands of years, and only optimized for writing computer code for a few decades at most.

The current frontier is halfway competent at benign closed 2D work, but still completely fumbles anything remotely close to open ended real world 3D work. It's getting better, but very slowly.

Re: Separating signal from noise in coding evaluations

#35

Post by Centigonal »

jheitmann wrote: post #32

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a…

If they fixed it, then it wouldn't be SWE-Bench Pro anymore, right? It'd be "SWE-Bench-Pro-Fixed-OpenAI." I think it's better optics for the independence of the benchmark if the OpenAI team lets some third party do the fixing and release the improved benchmark.

...Although OpenAI did exactly that when they released SWE-Bench Verified, so maybe I'm talking out of my butt here.

Re: Separating signal from noise in coding evaluations

#38

Post by tedsanders »

jheitmann wrote: post #32

It reads to me like "We did all the work you'd do to figure out how to fix the benchmark, then we decided to throw out the benchmark". Is there some reason the underlying data is so golden that it can't be patched? At the end they argue for a slightly more curated approach to benchmark generation, but my gut is that using messy ill-specified tests taken from real world data and patching them into fairness would be a…

Pointing out problems (e.g., hidden tests that assume narrow implementation details) is much easier than fixing them (e.g., creating tests that work for any possible choice of implementation).

Re: Separating signal from noise in coding evaluations

#39

Post by jaggederest »

carabiner wrote: post #23
warkdarrior wrote: post #18

Earlier quoted context omitted.

SWE-bench Verified was created to fix the problems of SWE-bench. Then SWE-Bench Pro was created because SWE-bench Verified had flaws. Now SWE-Bench Pro is shown to have flaws.

Is there a way to benchmark the accuracy, validity improvements in these successive benchmarks?

Bench Bench Pro Maxx Series S 360? The original Bench Bench Pro Maxx Series S had some quality issues, so that's the current followup. We've also released a higher order benchmark developed out of Bench Bench Pro Maxx Series S 360 One King Ranch edition, allowing future benchmark towers to be fully self-contained.

Re: Separating signal from noise in coding evaluations

#40

Post by kakugawa »

mlhpdx wrote: post #36

Fundamentally aren’t they concluding that tasks assigned to software developers (human or otherwise) are often incomplete, self contradictory or worse? This is the world in which their tool must play. I’m unsympathetic.

The more subtle point is that there's a gap between the task and its verification. e.g. if you have an open-ended / under-specified prompt, the verification needs to be able to handle all potential solutions.

So you can have a very narrow task prompt that's easy to verify (but likely too simple of a challenge). Or a more realistic task prompt that's much harder to verify. And likely harder to both build the robust verifier and run it cheaply.

Post reply on HN