Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

frontierharness.org

Post reply on HN
Print view View original

61–67 of 67 posts

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#62

Post by [deleted] »

vidarh wrote: post #8

Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't. Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops. Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that. Claiming on the…

[deleted]

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#63

Post by vidarh »

Edward40 wrote: post #50
vidarh wrote: post #8

Testing it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't. Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops. Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that. Claiming on the…

Good points! That’s a known limitation of our v1.0 benchmark with Claude Code. For v1.1, we're expanding both harnesses and models to evaluate the full harness × model matrix. This will highlight interaction effects (showing which harnesses suit which models best) and let us publish a full compatibility matrix. We shared more details on our roadmap in our blog post ( https://runta.com/blog/introducing-frontierharness…

That's great. It'd definitively be very interesting. Not least as an indicator of which harnesses to dig into the code of.

Kimi-cli is actually very interesting in that respect for the checkpoint / messaging mechanism I mentioned; it's basically almost like a model-initiated partial compaction of the end of a conversation to prune investigations that happened on the main agent loop - it'll be really interesting to see if it helps on any other models.

So oe thing that'd be really interesting to see when you expand to other models, would be if you mine the traces from those harnesses and generate stats on which tools get called. Seeing which models manage to take advantage of custom tools from which harnesses would be quite useful.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#64

Post by SSLy »

verdverm wrote: post #21
kakugawa wrote: post #16

With all the hype around the latest Gemini 3.8 Flash/Cyber release, will Antigravity CLI [1] be supported? 1/ https://antigravity.google/product/antigravity-cli

agy has fewer users than grok, both are ~1% based on some surveys I've seen, not every harness needs to be evaluated

hey, do you have a sample survey you could link? I don't have the slightest idea what I'd search for.
Post reply on HN