FrontierCode: An eval to measure whether you would actually merge the code

yep

At my day job, despite all the harnessing and providing extensive documentation and user stories via E2Es, I cannot trust models to deliver quality output. They are unable to, and reviewing 18 files of changes is the kind of work that increases my load and effort.

And yes, we have already split and optimized our documentation to not overwhelm the context.

In order to do this, the best flow is planning together, finding edge cases, having review skills, iterating, producing a business logic focused document describing the changes -> iterating to get a code changeset focused document.

Then I want to review step by step all the edits the model does.

On average this triplicates the amount of time required for a major change, but significantly improves business logic correctness and code quality, with the major benefit that it will require significantly less maintenance down the line and thus ends up being both a benefit on one side, and to improve harness on the other (more quality code, proper information, better examples for the models in the future).

The issue is: models are increasingly getting worse at this kind of work. While it is clear that they have better capabilities, the feedback loop has definitely degraded between Opus 4.7 and Opus 4.8, much more than it did between Opus 4.5 and 4.7.

This is very disappointing to me, as it is crystal clear that models are increasingly reinforced to deliver from prompt to the end result on their own and keep me more and more left out of the loop.

This has resulted in increasing frustration and makes my work slower, not better.

There is a chart that compares this in the article.

> You should compare apples to apples. Weight them in a way that factors in total task completion time as the measure of "effort", not the arbitrary effort settings provided by the AI company. I don't care what the underlying effort level is, I care which model out of multiple, if running for the same amount of time, completes my task to a more accurate degree.

That's your opinion, my goal is exactly what this benchmark measures, the end result being something I can merge into the codebase based on some configuration setup provided by the lab. I don't run 50 agents in parallel and I am able to use the $100 Anthropic plan just enough that I don't go over the limits.

Also what is your specific argument into the benchmark findings considering that some problems are solved by Opus and not solved by Codex? Whether one uses more tokens or not is a completely different metric.