FrontierCode: An eval to measure whether you would actually merge the code
1–10 of 67 posts
Re: FrontierCode: An eval to measure whether you would actually merge the code
#2:wave: i was on the team! AMA.
some headlines
- 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?"
- 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste.
- total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated and structured tasks with rubrics (even more work to turn tasks/prompts from devin-infra-specific to pluggable coding agent)
- results in 81% lower false positive rate than SWE-Bench Pro
- High quality bar: many QA stages & each task manually reviewed by Cognition researchers (examples in post)
Opus 4.8 scores 13% on FrontierCode Diamond.
one of my goals was also to datamine interesting stuff even on the easy tasks. for example, if you squint you can see the answer to "WTF Happened in late 2025" with coding models: https://x.com/swyx/status/2064081945567580323
Re: FrontierCode: An eval to measure whether you would actually merge the code
#3:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
How do you measure quality at scale ? Is there another model that determines if it adheres to codebase standard ?
Re: FrontierCode: An eval to measure whether you would actually merge the code
#4Since no one knows or can agree on what "code quality" is and we can't measure it for human output, I'm dubious about measuring it for LLMs
Re: FrontierCode: An eval to measure whether you would actually merge the code
#5[flagged]
Re: FrontierCode: An eval to measure whether you would actually merge the code
#6:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
How do you measure quality at scale ? Is there another model that determines if it adheres to codebase standard ?
see Beyond Unit Tests and Novel Grading Methods in TFA.
i think something like ~60% llm as judge rubrics and the rest as described. every rubric validated by maintainer. 3000 rubrics
Re: FrontierCode: An eval to measure whether you would actually merge the code
#7:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
Very cool! So glad to see people building and sharing evals that are better than SWE bench.
I'm curious - any particular reason you didn't put error bars on the graphs? Seems like it could be helpful when there are only 50 unique problems in the diamond set.
Re: FrontierCode: An eval to measure whether you would actually merge the code
#8> Today’s coding benchmarks have established that models can write correct code.
I wouldn't say that.
> But as AI-generated code becomes the dominant path to production
I really hope that's not the case.
Re: FrontierCode: An eval to measure whether you would actually merge the code
#9This looks great. Well reasoned, tons of work put into eval, thanks for building it.
It strikes me as kind of wild that good evals can drive tens to hundreds of millions of dollars of compute deployment in the wild — there’s something new and collaborative and competitive about the eval / frontier model race that’s quite interesting..
In this case “shorter actually mergable patches that open source maintainers would accept” feels like a great thing to deliver to the world.
I didn’t deep dive into good and bad patches, but I wonder if swyx or others on the team have predictions on saturation. Both when, and how useful will it be? That is, do you guys think this test is broad enough as written to get better behavior out of models, and if there is saturation on this test, will we see generalized better patch / coding behavior?
Re: FrontierCode: An eval to measure whether you would actually merge the code
#10:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
Very cool! So glad to see people building and sharing evals that are better than SWE bench. I'm curious - any particular reason you didn't put error bars on the graphs? Seems like it could be helpful when there are only 50 unique problems in the diamond set.
*50 unique problems but 20-40 rubrics per problem (something I had to keep reminding people internally who were unimpressed with the N)
simple answer is our reporting was pass@5. feel like you'd need like 50+ runs to have reasonable confidence intervals, which somehow i dont see other people do, so i also didnt insist on it.
hoping to work with to get this on their infra and evaluated along with whatever the industry standard is.