[deleted]
FrontierCode: An eval to measure whether you would actually merge the code
41–50 of 67 posts
Re: FrontierCode: An eval to measure whether you would actually merge the code
#42:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
> total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated and structured tasks with rubrics (even more work to turn tasks/prompts from devin-infra-specific to pluggable coding agent)
Heartening. We still haven’t automated making the world worse.
Re: FrontierCode: An eval to measure whether you would actually merge the code
#43Since no one knows or can agree on what "code quality" is and we can't measure it for human output, I'm dubious about measuring it for LLMs
You don't need universal consensus to measure something. There are many good quality measures of code quality.
> There are many good quality measures of code quality.
Besides the traditional and mechanical "less LOC", cyclomatic complexity and similar, which ones are you talking about exactly?
Re: FrontierCode: An eval to measure whether you would actually merge the code
#44YMMW but on my local tasks Opus 4.8 is nowhere near close to gpt 5.5. For the lack of a better word Opus is just soo damn lazy.
Re: FrontierCode: An eval to measure whether you would actually merge the code
#45:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
Does reporting each model at its best performing reasoning effort introduce a best-of-N/multiple-comparisons bias, especially if models have different numbers of effort levels?
Re: FrontierCode: An eval to measure whether you would actually merge the code
#46This isn't a fair way to chart this: "Each model is run 5 times at every available reasoning effort. For each effort, we average the metric across the 5 trials, then report each model’s score at its best performing reasoning level." For example, Anthropic's "medium" might involve 3x the amount of thinking and take 5x as long as OpenAI's idea of "medium". So now you've skewed all the results. It assumes that they're l…
I don't specifically care about Claude -vs- GPT, but comparing models at different amounts of test time compute is a gaping hole. It also means that any unreasonably-expensive token guzzling white-elephant model can top all the benchmarks and still be useless.
What we actually have is like a scaling law for test time compute, so it's silly to focus on specific Y values that someone benchmarked (at whatever default X values). Instead, characterize the slope or power of the scaling law, or just plot the damn curve for each model -vs- number of tokens or cost or something!
Noam Brown also raised this issue recently: https://x.com/polynoamial/status/2064210146558136827
Re: FrontierCode: An eval to measure whether you would actually merge the code
#47:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…
> total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated and structured tasks with rubrics (even more work to turn tasks/prompts from devin-infra-specific to pluggable coding agent) Heartening. We still haven’t automated making the world worse.
this gives my non tired self a chance to fix the typo:
- “ ON TOP of that, 40+ hours of real human work to turn…”
+ “ ON TOP of that, 40+ hours of real human work PER TASK to turn…”
Re: FrontierCode: An eval to measure whether you would actually merge the code
#48I'm liking the effort to make new, no-longer-saturated benchmarks. I'll also be a bit suspicious if some model aces it -- matching OSS maintainers' taste more often is a plausible improvement in quality but if they nail it every time they've been memorizing. Not saying FrontierCode should've done this, but benchmarking the interaction would be interesting. That is, if I get a diff with a blocking problem but writing…
i agree it would be interesting but apart from the fact that its be harder to measure and automate, theres real alpha in being the best truly async, hands off model/agent, which is what cog has been working on for 2 years now. its not that im opposed to steering or interaction mid task, its just that 1) it mostly Just Works, 2) it doesnt parallelize/scale well, 3) including on proactive agents (https://docs.devin.ai/product-guides/automations).
see my “semi async valley of death” post. people are pursuing both sides but per bitter lesson only one side scales indefinitely with compute
that said, multistage rollouts and synhetic rubrics (using grpo advantage? see dr tulu paper) somewhat approximate human intervention and interaction, so theres known ways to model that, its just not thaaaat valuable
Re: FrontierCode: An eval to measure whether you would actually merge the code
#49[deleted]
Re: FrontierCode: An eval to measure whether you would actually merge the code
#50Earlier quoted context omitted.
thanks - credit to silas, eric, ben, and team for the depth of the evals, and the rest of the research team for doing the transcript reading parties lol by nature of being based on open source, frontiercode public will saturate very very quickly. frontiercode main will be >80% in less than a year. hopefully diamond will last a bit longer. we can do annual refreshes, thats not my strategy for staying relevant - what i…
Interesting. So frontiercode-IBM-Diamond is a thing you’d hope to sell the creation of and certification of? And if it’s published then you’d expect model providers to train to forntiercode-IBM-Pro or whatever and publish it so that it would be considered a good model to use inside IBM? (Obviously just a random corporate choice here).
no, single customer focus would be bad for a number of reasons.
but frontiercode-finance? thatd be cool…