Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

frontierharness.org

Post reply on HN
Print view View original

41–50 of 67 posts

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#41

Post by try-working »

monster_truck wrote: post #5

It's nice to see time reflected here. Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing…

DeepSeek is consistently the fastest model, and Kimi the slowest. GPT in the middle. Anthropic is probably on the slow side as well.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#42

Post by infecto »

dfltr wrote: post #33
nsingh2 wrote: post #29

One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…

It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it. It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"

Not weird or odd in the least. I think it clearly shows that Pi barebones at least in this set of tests, preformed better and cheaper cost that OMP.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#43

Post by infecto »

nsingh2 wrote: post #29

One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…

I use Pi barebones.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#44

Post by nijave »

vidarh wrote: post #8

Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't. Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops. Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that. Claiming on the…

Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.

Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls

There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#45

Post by joshheitzman »

I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#46

Post by nijave »

monster_truck wrote: post #5

It's nice to see time reflected here. Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing…

>nobody is going to be doing 17 benchmark passes on a $50-100 task

Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#47

Post by nijave »

try-working wrote: post #41
monster_truck wrote: post #5

It's nice to see time reflected here. Deepseek is fuckin fast! Seems like the bigger the task, the faster it gets, which is kind of unfortunate because nobody is going to be doing 17 benchmark passes on a $50-100 task. I'm assuming the brief pause before it avalanches out 16kb of text at upwards of 250tk/s (multiples beyond anything resembling a comfortable reading speed) is some sort of workload evaluator directing…

DeepSeek is consistently the fastest model, and Kimi the slowest. GPT in the middle. Anthropic is probably on the slow side as well.

Agree with the exception of Haiku. Haiku is insanely fast if it works for your task

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#48

Post by Edward40 »

joshheitzman wrote: post #45

I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.

We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#50

Post by Edward40 »

vidarh wrote: post #8

Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't. Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops. Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that. Claiming on the…

Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post (https://runta.com/blog/introducing-frontierharness-eval/).
Post reply on HN