Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FrontierCode: An eval to measure whether you would actually merge the code

cognition.ai

Post reply on HN
Print view View original

11–20 of 67 posts

Re: FrontierCode: An eval to measure whether you would actually merge the code

#11

Post by swyx »

vessenes wrote: post #9

This looks great. Well reasoned, tons of work put into eval, thanks for building it. It strikes me as kind of wild that good evals can drive tens to hundreds of millions of dollars of compute deployment in the wild — there’s something new and collaborative and competitive about the eval / frontier model race that’s quite interesting.. In this case “shorter actually mergable patches that open source maintainers would…

thanks - credit to silas, eric, ben, and team for the depth of the evals, and the rest of the research team for doing the transcript reading parties lol

by nature of being based on open source, frontiercode public will saturate very very quickly. frontiercode main will be >80% in less than a year. hopefully diamond will last a bit longer. we can do annual refreshes, thats not my strategy for staying relevant - what i'm more excited to get funding for is private held out version of frontiercode based on repros of real enterprise customer problems. in an ideal agent lab (https://latent.space/p/agent-labs) you meticulously build up this domain understanding and that is essentially why both model labs and serious customers come to you.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#12

Post by zakisaad »

einpoklum wrote: post #8

> Today’s coding benchmarks have established that models can write correct code. I wouldn't say that. > But as AI-generated code becomes the dominant path to production I really hope that's not the case.

How do you define "correct" code?

Re: FrontierCode: An eval to measure whether you would actually merge the code

#13

Post by typs »

swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

What did you do around cross-harness testing? I don't see anything in the blog post about what harnesses were used in evaluation. SOTA benchmarks have consistently shown that frontier model performance is quite sensitive to what tools are exposed (e.g. str_replace vs. apply_patch) as the labs are RLing on their own harnesses. Did you do testing of the models in a standard setup or in their native harnesses?

Re: FrontierCode: An eval to measure whether you would actually merge the code

#14

Post by Topfi »

Great effort and a bit closer to my private evals than DeepSWE. I greatly appreciate the focus on false negative and positives, along with simply being far more focused on actual, mergeable quality output over plain passing. Could see a lot of others adopt your list of metrics as a basis, they are very well defined and solid coverage of everything one should want out of code provided, not just focused on one or two narrow targets. Will incorporate a lot of these ideas in my own tests and polish some other parts where I somewhat unintentionally already went into a roughly similar direction.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#15

Post by swyx »

typs wrote: post #13
swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

What did you do around cross-harness testing? I don't see anything in the blog post about what harnesses were used in evaluation. SOTA benchmarks have consistently shown that frontier model performance is quite sensitive to what tools are exposed (e.g. str_replace vs. apply_patch) as the labs are RLing on their own harnesses. Did you do testing of the models in a standard setup or in their native harnesses?

yes well aware :) numbers shown are on "house" harnesses eg codex with gpt and claude code with opus.

fwiw we have examples of each model doing better on NON-house harnesses too - speaking jsut for myself i think the "the labs are RLing on their own harnesses" narrative is kinda overstated if you think through wanting to have any meaningful api business (often eg the labs will give guidance on what is prefered and the agent labs can easily match tool contract to that, which is to say, the "home turf advantage" isnt as large as you think it is if you try a little bit)

Re: FrontierCode: An eval to measure whether you would actually merge the code

#16

Post by tedsanders »

swyx wrote: post #10
tedsanders wrote: post #7

Earlier quoted context omitted.

Very cool! So glad to see people building and sharing evals that are better than SWE bench. I'm curious - any particular reason you didn't put error bars on the graphs? Seems like it could be helpful when there are only 50 unique problems in the diamond set.

*50 unique problems but 20-40 rubrics per problem (something I had to keep reminding people internally who were unimpressed with the N) simple answer is our reporting was pass@5. feel like you'd need like 50+ runs to have reasonable confidence intervals, which somehow i dont see other people do, so i also didnt insist on it. hoping to work with to get this on their infra and evaluated along with whatever the industry…

Makes sense, thanks. I suppose error bars are tricky if trying to handle problem-to-problem variance, rubric-to-rubric variance, and run-to-run variance all at once.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#17

Post by chris_st »

swyx wrote: post #15
typs wrote: post #13

Earlier quoted context omitted.

What did you do around cross-harness testing? I don't see anything in the blog post about what harnesses were used in evaluation. SOTA benchmarks have consistently shown that frontier model performance is quite sensitive to what tools are exposed (e.g. str_replace vs. apply_patch) as the labs are RLing on their own harnesses. Did you do testing of the models in a standard setup or in their native harnesses?

yes well aware :) numbers shown are on "house" harnesses eg codex with gpt and claude code with opus. fwiw we have examples of each model doing better on NON-house harnesses too - speaking jsut for myself i think the "the labs are RLing on their own harnesses" narrative is kinda overstated if you think through wanting to have any meaningful api business (often eg the labs will give guidance on what is prefered and th…

What "non-house" harnesses have you found to work best?

Re: FrontierCode: An eval to measure whether you would actually merge the code

#18

Post by newsicanuse »

zakisaad wrote: post #12
einpoklum wrote: post #8

> Today’s coding benchmarks have established that models can write correct code. I wouldn't say that. > But as AI-generated code becomes the dominant path to production I really hope that's not the case.

How do you define "correct" code?

The code that gets stuff done instead of beating around the bush making unxpected errors

Re: FrontierCode: An eval to measure whether you would actually merge the code

#19

Post by kube-system »

singpolyma3 wrote: post #4

Since no one knows or can agree on what "code quality" is and we can't measure it for human output, I'm dubious about measuring it for LLMs

You don't need universal consensus to measure something. There are many good quality measures of code quality.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#20

Post by glerk »

swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

This looks really great, more thoughtful than any benchmark that I've seen until now!

I'm curious if you're only interested in scoring frontier models or you would accept submission from custom harnesses? I am working on multi-model harnesses and would love to test them against your benchmark. Do you plan on releasing the tasks publicly?

Post reply on HN