Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

SWE-bench Verified no longer measures frontier coding capabilities

openai.com

Post reply on HN
Print view View original

191–200 of 209 posts

Re: SWE-bench Verified no longer measures frontier coding capabilities

#191

Post by falcor84 »

anthonypasq96 wrote: post #160
jjmarr wrote: post #133

Earlier quoted context omitted.

I'm making an LLM agent that can play DS games. The biggest blocker is clicking on the right spot to move things around in space rather than reasoning abilities. Arc AGI seems to test that as well. Every game is a rectangular grid to make it as easy as possible yet the AIs still fail. I'm fairly certain the way forward isn't through agents directly interfacing with UIs but through agents using scripts and other tools…

the whole point of arc-agi 3 is that if models are AGI then they should be able to solve the same tasks as humans do given the same information, but they cant. allowing scripts and harnesses and whatnot completely defeats the purpose.

But humans aren't just a "reasoning component"; our nervous system (and body in general) provides us with significant capabilities that would be considered a "harness" for our frontal lobe. It just seems silly to me to try to solve all of this in a single leap. But I guess that they just feel burned by how relatively quickly ARC-AGI 2 was solved

Re: SWE-bench Verified no longer measures frontier coding capabilities

#192

Post by orangebread »

euleriancon wrote: post #170
orangebread wrote: post #53

Earlier quoted context omitted.

Wow. This benchmark definitely feels more accurate than the other rankings I've seen. My experience with gpt 5.4/5.5 is that they are technically flawless and if there are any technical issues that is because the input didn't provide enough clarity; that's not to say that it doesn't autonomously react to any issues during bug fixes or implementations, but it'll tend to nail its tasks without leaving behind gaps. Opus…

Are we looking at the same data? On that site I see that opus 4.7's and gpt 5.5's g scores are within each others confidence intervals, and both significantly ahead of the number 3 model. Your comment makes it sound like they are miles apart, which the benchmark doesn't seem to support. Edit: I looked at the data more and the two models are only basically equal when looking at the mean of all the tests. Gpt 5.5 signi…

Right, I'm including my own observations in what the leaderboard is showing. Could be confirmation bias, but I use both Opus and GPT extensively and since GPT 5.4 I have noticed that Opus doesn't even begin to touch GPT's level of technical depth. I was hoping Opus 4.7 would close that gap, but unfortunately it doesn't even compare to GPT 5.4 in that sense.

I'm not being a hater, I love Opus for different reasons, but I can't rely on it for its technical ability.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#193

Post by civvv »

mxwsn wrote: post #96
civvv wrote: post #92

Earlier quoted context omitted.

This is likely true. I think model quality has stagnated and that its likely a non-trivial task to find a new improvement vector. Scaling the width of the model (which has been the driving force behind the speed of improvement thus far) seems to have reached its limit. It will be interesting to see the implications of this. Tooling can only do so much in the long term.

How do you know that width scaling has been the driving force of improvement?

I am no insider and have never even tried to build an LLM, so I can only guess. But the general sentiment seems to be that this is the case. If you are interested, I would recommend you read the MIT paper "Superposition Yields Robust Neural Scaling" [0]. It confirms an interesting trend: models represent more features/concepts than they have clean independent dimensions, so features overlap. Increasing model dimension reduces this geometric interference, which lowers loss in a predictable way, but with diminishing returns.

This has, in my opinion, likely been the primary vector in getting better models thus far, but MIT mathematically proves that it yields diminishing returns for each new dimension added. It will get more and more expensive and the cost-return will or probably already has made it infeasible.

Ilya appear to support sentiment this as well. [1]

[0] - https://openreview.net/forum?id=knPz7gtjPW [1] - https://www.businessinsider.com/openai-cofounder-ilya-sutske...

Re: SWE-bench Verified no longer measures frontier coding capabilities

#195

Post by girvo »

red75prime wrote: post #155
cptskippy wrote: post #149

Earlier quoted context omitted.

He said complicated code bases. LLMs are great at producing small snippets of code to address very targeted problems.

Great on small snippets of code, passable on larger pieces of code, great at finding vulnerabilities in large pieces of code, terrible in Zork. All-in-all, a jagged frontier that defies a simple sarcastic characterization.

Very kiki, not very bouba, as Aphyr rightfully stated.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#197

Post by mnky9800n »

CamperBob2 wrote: post #77
mnky9800n wrote: post #29

Earlier quoted context omitted.

This is why I made Zork bench. Zork, the text adventure game, is in the training data for LLMs. It’s also deterministic. Therefore it should be easy for an LLM to play and complete. Yet they don’t. Understanding why is the goal of Zork bench. https://github.com/mnky9800n/zork-bench

Actually the Zorks weren't deterministic, especially Zork II. The Wizard could F you over pretty badly if he appeared at an inopportune time.

I feel like you are being pedantic. There are very few parts of Zork that are not static to the game. Yes the thief shows up randomly but that’s not the main point of the game.

Re: SWE-bench Verified no longer measures frontier coding capabilities

#198

Post by jayd16 »

Jcampuzano2 wrote: post #18

Its pretty clear that any benchmark that comes out will be outdated and exist within the training data with short measure. There will always be an incentive to optimize specifically for these benchmarks even if just for marketing material. Sure there is a training cutoff, but its usually only 3-6 months off of the public release dates. The problem with coding benchmarks then becomes creating novel benchmarks that are…

Lets just base benchmarks on bounty rankings. To bench a model, you have it look at PRs on some open source projects. It has to complete a novel task or improve a previous task but no points for just re-doing a task with an existing PR. We rank the tasks by difficulty for the benchmark post-facto, once completed.

If an AI company wants to show off, it'll have to crush some OSS PRs. If another company wants to say their model remains supreme, it'll have to complete other tasks that were left on the table.

Of course, you would only bother the OSS project with new PRs once you were actually not embarrassed by what your model did.

In this way, rankings are created from jolly combat and one-ups-manship and we get some OSS work done.

(mostly joking but it would be a fun way to do things)

Re: SWE-bench Verified no longer measures frontier coding capabilities

#199

Post by gertlabs »

yalok wrote: post #180
gertlabs wrote: post #47

A better benchmark needs to be objectively scored, have multi-disciplinary, breadth, and be scalable (no single correct answer). That's what we designed at https://gertlabs.com . We put a lot of thought into it, and kept it mostly (not fully) related to problem solving through coding.

amazing to see Claude Code top models still way above all other models for C++ & Java, while GPT 5.5 is higher in Python & JS and others. Shows the skew in the training data sets, and maybe the go-to-market focus - with Anthropic focusing on enterprise customers much more than OpenAI? Matches with my experience with Opus for C++. C# results are empty - @gertlabs - any ETA for those?

C# testing is a new feature added a few days ago from HN comment suggestions, samples will continue growing. Most C# data is currently for non-agentic workloads: https://gertlabs.com/?mode=oneshot_coding

Re: SWE-bench Verified no longer measures frontier coding capabilities

#200

Post by CamperBob2 »

Schlagbohrer wrote: post #185
CamperBob2 wrote: post #77

Earlier quoted context omitted.

Actually the Zorks weren't deterministic, especially Zork II. The Wizard could F you over pretty badly if he appeared at an inopportune time.

Was that using an RNG? Or is the entire game deterministic?

It used an RNG. The usual practice back then was to spin a counter while waiting for keypresses, so that might affect the question when dealing with an external harness, I suppose.
Post reply on HN