Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

FrontierCode: An eval to measure whether you would actually merge the code

cognition.ai

Post reply on HN
Print view View original

51–60 of 67 posts

Re: FrontierCode: An eval to measure whether you would actually merge the code

#51

Post by swyx »

ssivark wrote: post #46
nullbio wrote: post #24

This isn't a fair way to chart this: "Each model is run 5 times at every available reasoning effort. For each effort, we average the metric across the 5 trials, then report each model’s score at its best performing reasoning level." For example, Anthropic's "medium" might involve 3x the amount of thinking and take 5x as long as OpenAI's idea of "medium". So now you've skewed all the results. It assumes that they're l…

I don't specifically care about Claude -vs- GPT, but comparing models at different amounts of test time compute is a gaping hole. It also means that any unreasonably-expensive token guzzling white-elephant model can top all the benchmarks and still be useless. What we actually have is like a scaling law for test time compute, so it's silly to focus on specific Y values that someone benchmarked (at whatever default X…

ok i mean i agree, how is it a gaping hole when its literally the second (and third and fourth..) chart on the post? yes token cost and reasoning efficiency is important, hence the 2D pareto charts

Re: FrontierCode: An eval to measure whether you would actually merge the code

#52

Post by swyx »

llama_drama wrote: post #45
swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

Does reporting each model at its best performing reasoning effort introduce a best-of-N/multiple-comparisons bias, especially if models have different numbers of effort levels?

to you it may do idk. note that if you scroll past fig 1 you get into a nice data explorer that breaks out pass@5 by reasoning level with token and $ and step cost visualized. i think some other commenters on this hn thread got very worked up about stuff we actually agree on.

internally ive charted everything and am satisfied that theres no meaningful rank bias introduced. weve sliced it every which way. in fact we have not even published the best looking charts for this story to be told, because we have further publishing plans on frontiercode

tldr “trust me bro” this isnt the issue and if anything we couldve done more to increase N as tedsanders below points out

Re: FrontierCode: An eval to measure whether you would actually merge the code

#53

Post by VulgarExigency »

swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

Any chance of also benchmarking a couple of more affordable Chinese models? (specifically Deepseek and Xiaomi's MiMo)

Re: FrontierCode: An eval to measure whether you would actually merge the code

#54

Post by swyx »

VulgarExigency wrote: post #53
swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

Any chance of also benchmarking a couple of more affordable Chinese models? (specifically Deepseek and Xiaomi's MiMo)

i think will help us do that best on their standardized model matrix. for frontiercode’s launch we were focused on.. the frontier models

Re: FrontierCode: An eval to measure whether you would actually merge the code

#55

Post by swyx »

fouc wrote: post #28
swyx wrote: post #2

:wave: i was on the team! AMA. some headlines - 3000 rubrics on code quality. First benchmark to measure: "would this code get actually merged?" - 20+ expert open-source maintainer created tasks on their own repos to capture their opinion & taste. - total 1000+ hours of real life software maintainer work captured in dataset. ON TOP of that, 40+ hours of real human work to turn that real life work into well validated…

I'm a bit disappointed that Opus 4.6 wasn't in this because the tokenizer changed quite a bit from 4.7 onward. I was so annoyed by 4.7 that I've been forcing 4.6 ever since. I've been annoyed by 4.8 a bit too, so I haven't felt the urge to move on.

shared older model numbers here https://www.latent.space/p/ainews-frontiercode-benchmarking

tldr theres been broad progress despite your observed regressions

Re: FrontierCode: An eval to measure whether you would actually merge the code

#56

Post by maccard »

vanuatu wrote: post #29
newsicanuse wrote: post #18

Earlier quoted context omitted.

The code that gets stuff done instead of beating around the bush making unxpected errors

i suspect this is highly dependent on what you're working on from my experience if you give the models a way to self-verify correctness they succeed basically 100% of the time

> from my experience if you give the models a way to self-verify correctness they succeed basically 100% of the time

My experience is that if you can get the model to one shot the task, you'll do fine but if it has to iterate it leaves things worse than before and almost always requires human intervention after burning through an enormous amount of tokens

Re: FrontierCode: An eval to measure whether you would actually merge the code

#57

Post by VulgarExigency »

swyx wrote: post #54
VulgarExigency wrote: post #53

Earlier quoted context omitted.

Any chance of also benchmarking a couple of more affordable Chinese models? (specifically Deepseek and Xiaomi's MiMo)

i think will help us do that best on their standardized model matrix. for frontiercode’s launch we were focused on.. the frontier models

What qualifies as a frontier model? From my personal "taste tests", I wouldn't have placed Sonnet or Kimi above Deepseek Pro or MiMo, or Gemini 3.1 Flash Lite above Deepseek Flash, but they're listed in the benchmark.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#58

Post by vessenes »

swyx wrote: post #50
vessenes wrote: post #34

Earlier quoted context omitted.

Interesting. So frontiercode-IBM-Diamond is a thing you’d hope to sell the creation of and certification of? And if it’s published then you’d expect model providers to train to forntiercode-IBM-Pro or whatever and publish it so that it would be considered a good model to use inside IBM? (Obviously just a random corporate choice here).

no, single customer focus would be bad for a number of reasons. but frontiercode-finance? thatd be cool…

Ahh interesting. There are companies making good money doing private equity dd work, largely custom harnesses. A lot of open space right now.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#59

Post by kube-system »

embedding-shape wrote: post #43
kube-system wrote: post #19

Earlier quoted context omitted.

You don't need universal consensus to measure something. There are many good quality measures of code quality.

> There are many good quality measures of code quality. Besides the traditional and mechanical "less LOC", cyclomatic complexity and similar, which ones are you talking about exactly?

I'm not talking about any specific measure. There are many. Pick some, and you have a benchmark.

Re: FrontierCode: An eval to measure whether you would actually merge the code

#60

Post by [deleted] »

swyx wrote: post #51
ssivark wrote: post #46

Earlier quoted context omitted.

I don't specifically care about Claude -vs- GPT, but comparing models at different amounts of test time compute is a gaping hole. It also means that any unreasonably-expensive token guzzling white-elephant model can top all the benchmarks and still be useless. What we actually have is like a scaling law for test time compute, so it's silly to focus on specific Y values that someone benchmarked (at whatever default X…

ok i mean i agree, how is it a gaping hole when its literally the second (and third and fourth..) chart on the post? yes token cost and reasoning efficiency is important, hence the 2D pareto charts

[deleted]
Post reply on HN