Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

61–70 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#61

Post by rustyhancock »

I think an Olympiad format is better. But the financial incentive is such that it might be near impossible to stop leaks.

I.e. A panel comes up with a series of problems.

Like advent of code or project Euler but more complex and constricted.

Benchmark outcomes could be performance points and measure of cost, time to solution (well token count really).

A couple times per year it's run.

It avoids overfitting.

Overtime the tasks can become more complex if needed.

If they benchmax it into being able to complete full products from spec and robust implementations amazing.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#62

Post by motoboi »

vintagedave wrote: post #20
motoboi wrote: post #16

Earlier quoted context omitted.

It’s saying that 16% of the problems have well, problems.

You're right - I did not apply the math. (I won't edit, in order to let the parent comment still make sense, and thankyou for the correction.) So not one in four, but one in six problems have problems. That is extraordinarily high and the point still stands: is this truly saying a [large proportion] of the questions and answers were wrong, this whole time, and if so how was it ever a valid measurement?

Wait until you discover how many wrong labeled images in imagenet and that it still kickstarted the deeplearning revolution.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#63

Post by cowartc »

The headline leads with contamination, but buried is that 59% of audited failures had test design defects. That's a measurement system never validated against ground truth before being adopted industry-wide as a score that mattered. They reported on it for two years but the gauge was broken the entire time.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#64

Post by therealdrag0 »

threepts wrote: post #31

Why don't they ask their premier model to generate a bench for them? Jokes aside, a benchmark I look forward to is ARC-AGI-3. I tried out their human simulation, and it feels very reasoning heavy. Leaderboard: https://arcprize.org/leaderboard (Most premier models don't even pass 5 percent.)

[dead]

Re: SWE-bench Verified no longer measures frontier coding capabilities

#65

Post by nothinkjustai »

cowartc wrote: post #63

The headline leads with contamination, but buried is that 59% of audited failures had test design defects. That's a measurement system never validated against ground truth before being adopted industry-wide as a score that mattered. They reported on it for two years but the gauge was broken the entire time.

Ai comments are banned here.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#66

Post by sowbug »

threepts wrote: post #31

Why don't they ask their premier model to generate a bench for them? Jokes aside, a benchmark I look forward to is ARC-AGI-3. I tried out their human simulation, and it feels very reasoning heavy. Leaderboard: https://arcprize.org/leaderboard (Most premier models don't even pass 5 percent.)

Why don't they ask their premier model to generate a bench for them?

It's not a crazy idea. Have the older model interview the newer one and then ask both (or maybe a third referee model) which one they think is smarter. Repeat 100x with different seeds. The percentage of times both sides agree the newer model won is the score.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#68

Post by xtracto »

threepts wrote: post #31

Why don't they ask their premier model to generate a bench for them? Jokes aside, a benchmark I look forward to is ARC-AGI-3. I tried out their human simulation, and it feels very reasoning heavy. Leaderboard: https://arcprize.org/leaderboard (Most premier models don't even pass 5 percent.)

Can AI write a problem so difficult that even AI cannot solve?

Hehe

Re: SWE-bench Verified no longer measures frontier coding capabilities

#69

Post by roenxi »

WarmWash wrote: post #46
mnky9800n wrote: post #29

Earlier quoted context omitted.

This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench

The open models only give the SOTA models a run for their money on gameable benchmarks. On the semi-private ARC-AGI 2 sets they do absolutely awfully ( It might be too expensive, but I would be interested in the benchmarks for the current crop of SOTA models.

Have the open models been tried? When I look at the leaderboard [0] the only qwen model I see is 235B-A22B. I wouldn't expect an MoE model to do particularly well, from what I've seen (thinking mainly of a leaderboard trying to measure EQ [1]) MoE models are at a distinct disadvantage to regular models when it comes to complex tasks that aren't software benchmark targets.

[0] https://arcprize.org/leaderboard

[1] https://eqbench.com/index.html

Post reply on HN