[flagged]
SWE-bench Verified no longer measures frontier coding capabilities
161–170 of 209 posts
Re: SWE-bench Verified no longer measures frontier coding capabilities
#162Translation: Now that all rest sets are ingested, we need to move the bar that gave use several years of free PR.
See also: https://this.os.isfine.org/blog/posts/us-ai-labs-love-the-ai...
Re: SWE-bench Verified no longer measures frontier coding capabilities
#163Earlier quoted context omitted.
> 93.6% (congrats Anthropic) But the article says "We audited a 27.6% subset of the dataset that models often failed to solve [which is 19.1% of the problems at time of publication] and found that at least 59.4% of the audited problems have flawed test cases that reject functionally correct submission" 0.191 * 0.594 > 1 - 0.936 Does this mean that the audited subset wasn't representative? Or that Anthropic is getting…
I suggest reading the Mythos report's discussion on SWE-bench and contamination. I think it's fairly convincing that you can account for contamination and still trust SWE-bench numbers on models that aren't over-optimized for it.
You can trust that a model that scores 40% vs a model that scores 90% is indeed worse.
You can’t trust it that a model that scores 93% is better at software engineering than a model that scores 90%, because at that point it’s impossible to distinguish between recall and reasoning.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#164Earlier quoted context omitted.
I have worked on similar problems. See e.g. [1]. The LLMs I have tested have terrible world models and intuitions for how actions change the environment. They're also not great at discerning and pursuing the right goals. They're like an infinitely patient five-year old with amazing vocabulary. [1]: https://entropicthoughts.com/updated-llm-benchmark (more descriptions available in earlier evaluations referenced from t…
You can code your prompts to read and write an external world model on the side. This is what most people do who are seriously doing games with LLMs.
What do you mean with this? What is this world model, what does it capture?
Re: SWE-bench Verified no longer measures frontier coding capabilities
#165Earlier quoted context omitted.
I'm making an LLM agent that can play DS games. The biggest blocker is clicking on the right spot to move things around in space rather than reasoning abilities. Arc AGI seems to test that as well. Every game is a rectangular grid to make it as easy as possible yet the AIs still fail. I'm fairly certain the way forward isn't through agents directly interfacing with UIs but through agents using scripts and other tools…
the whole point of arc-agi 3 is that if models are AGI then they should be able to solve the same tasks as humans do given the same information, but they cant. allowing scripts and harnesses and whatnot completely defeats the purpose.
Humans haven't interacted with computers by typing in "5 columns right, 3 columns down" since before I was born. They use a mouse and keyboard.
Meanwhile AI agents are expected to guess pixels and fail each time.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#166It was never that great, it seems. For all of 2025 there was virtually no improvement in the rate at which models produced quality code. They only got better at passing automated tests. https://entropicthoughts.com/no-swe-bench-improvement
It's not true that there was no improvement in the rate at which models produced quality code.
Jan 2025 was Claude 3.5 Sonnet, Gemini 1.5 Pro and OpenAI had GPT-4o.
As someone who used all those models, as well as today's frontier models - today's models are a significant step up from those.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#167Earlier quoted context omitted.
Those who fail to study history (or live through it) are doomed to repeat it. SPECint and SPECfp went through this exact movie: benchmark, saturate, retire, replace, repeat. The treadmill is the product. I don't have the solution just noticing the pattern.
That's a slightly different problem. There's no thing as saturation for a performance benchmark like SPEC; we can always conceive of a faster processor (even if we don't know how to build one). Saturation is the problem that once you are at (or near) 100% pass rate on a test of pass/fail questions, there's no room for the score to keep going up and the test has lost any power to discriminate between competing options…
Maybe OP was thinking about compilers "cracking" certain SPEC benchmarks: implementing exactly the optimization needed to boost a benchmark quite a lot, but that opt. probably won't apply to any other code out there (usually it's so targeted and risky with general C/C++ code that intentionally it doesn't work on anything else). That happened a couple of times over the years, I know about the Intel compiler cases for ex. I can certainly see LLM providers adding tricks that help a certain class of benchmarks, but doesn't help much for anything else.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#168Earlier quoted context omitted.
How do you know that width scaling has been the driving force of improvement?
I mean, it's not exactly a PhD level question. One can infer from the extreme demand of GPUs and DRAM + new data center construction that all the providers are banking on width.
No? That could just be fomo, actual adoption, or a number of other things.
Re: SWE-bench Verified no longer measures frontier coding capabilities
#169[dead]
Re: SWE-bench Verified no longer measures frontier coding capabilities
#170A better benchmark needs to be objectively scored, have multi-disciplinary, breadth, and be scalable (no single correct answer). That's what we designed at https://gertlabs.com . We put a lot of thought into it, and kept it mostly (not fully) related to problem solving through coding.
Wow. This benchmark definitely feels more accurate than the other rankings I've seen. My experience with gpt 5.4/5.5 is that they are technically flawless and if there are any technical issues that is because the input didn't provide enough clarity; that's not to say that it doesn't autonomously react to any issues during bug fixes or implementations, but it'll tend to nail its tasks without leaving behind gaps. Opus…
Are we looking at the same data? On that site I see that opus 4.7's and gpt 5.5's g scores are within each others confidence intervals, and both significantly ahead of the number 3 model.
Your comment makes it sound like they are miles apart, which the benchmark doesn't seem to support.
Edit: I looked at the data more and the two models are only basically equal when looking at the mean of all the tests. Gpt 5.5 significantly outperforms opus 4.7 in coding, while opus 4.7 significantly outperforms in "decision making." I'm not seeing details on what decision making explicitly means.