31–40 of 49 posts
Interesting benchmark. I can't help but notice that they're benchmarking Opus 4.6 (Anthropic's latest and greatest model) against GPT-5.2 (which is three generations behind OpenAI's latest coding models: GPT-5.2-Codex, GPT-5.3-Codex and the latest GPT-5.4).
5.2 and 5.2 Codex is arguably the same gen.
Really long-term task benchmark showing significant improvements in very recent models, while also showing really bad regression rates across the board.
Uh, Opus 4.6 avoids introducing regressions 75% of the time?
We are talking about regressions, what once worked no longer does, and should be measured in 9s
Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…
It's the other way around - Claude Code is the proprietary one. Codex CLI is open source: https://github.com/openai/codex You can definitely access the latest models via the API. That's how Codex CLI works.
But it was in codex
Claude wins by a large margin * Claude Opus 4.6 : 0.71 * Claude Opus 4.5 : 0.51 * KIMI-K2.5 : 0.37 * GLM-5 : 0.36 * GPT-5.2 : 0.23 Note: later GPT versions seem to be only available within openAi's proprietary codex cli, so can't be tested - and if tested via the codex cli "harness" it wouldn't be a pure model-to-model comparison any more. --- Of course, the interesting follow-up question is: How well perform these m…
But the interesting comparison when evaluating coding agent capabilities is to evaluate the offerings given to users.
So this means comparing Claude Code to Codex to whatever CLI tools Kimi, GLM, and others give you.
And it might mean throwing Cursor, OpenCode, Amp, Pi, mini-swe-agent, etc into the mix
Something hard to capture in benchmarks: project-level conventions. A well-maintained CLAUDE.md at the repo root — describing architecture, naming patterns, test conventions — gives the agent context it internalizes before touching code. My regression rate dropped noticeably once I started maintaining that kind of project metadata. Model choice is only half the equation — the other half is how well you've structured the information environment the agent works in.
Earlier quoted context omitted.
Uh, Opus 4.6 avoids introducing regressions 75% of the time?
So 1/4 times it does not introduce a regression. That's still pretty bad imo. If 1/4 commits introduced regressions, what would your team do? We are talking about regressions , what once worked no longer does, and should be measured in 9s
[flagged]
This may also be the limit to the quality of an automated port to another language. What isn't encoded as automated tests or manual test procedure cannot be verified.
So often I'm amazed at what it's possible to accomplish from a prompt that's certainly insufficient with insufficient context. "It should have been necessary to specify more context there," or "I would have thought that it wasn't possible to do that without reading in more context than just one source code file," and then a few prompts later, "there's where we failed for trying to skimp on context"
To prevent architectural rework as a human developer also requires substantial ahead-of-time codebase review.
Are AGENTS.md files the best place to summarize more comprehensive codebase review and useful dense context like guidelines for testing and architectural components in order to avoid rework?