Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

frontierharness.org

Post reply on HN
Print view View original

51–60 of 67 posts

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#51

Post by nijave »

jonstewart wrote: post #9

Forgive my stupidity but how do you run Claude Code with non-Anthropic models?

I use routatic/proxy (actually my own fork with various changes/fixes http://github.com/nijave/proxy/ -- fyi the fork is all vibes...)

Some gateways/proxies/hosts also offer multiple endpoints (z.ai has both Anthropic endpoint and Completions endpoint)

Edit: Here's the PoC I made for work although it assumes you're using Cloudflare Access (VPN) with a Cloudflare AI Gateway to hand wave the auth away (auth to the LLM API gets auto injected by Cloudflare Access derived from the client's VPN connection) https://gist.github.com/nijave/6f5e5793a992619808135ce85fa49...

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#52

Post by joshheitzman »

Edward40 wrote: post #48
joshheitzman wrote: post #45

I was excited until I saw cost was only provided as the median. Your provider will bill you for all your tasks and one can get back to that total from the mean by multiplying by the number of tasks. This isn't possible with the median and I suspect the median is likely below the mean so this understates the actual costs.

We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.

Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems.

I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one specific task for each harness (or the mean of two it looks like in the case of Exo Harness; I couldn't spot the one task that lined up with the headline cost), but the same task isn't used for the headline cost for each harness. It's not the same as picking a task at random to use as the headline task, but its in the ballpark.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#53

Post by Edward40 »

markbao wrote: post #32

This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed. People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up. Thanks for making this and filling a real gap!

Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#54

Post by Edward40 »

infecto wrote: post #43
nsingh2 wrote: post #29

One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…

I use Pi barebones.

I also prefer using vanilla Pi over Oh My Pi.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#55

Post by Edward40 »

nijave wrote: post #49

It would be interesting to try to control for system prompt although obviously there's some coupling to what the harness provides and the instructions the model receives on using it

We'd rather not mess with system prompts, we just eval them as shipped to keep the results reproducible.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#56

Post by Edward40 »

joshheitzman wrote: post #52
Edward40 wrote: post #48

Earlier quoted context omitted.

We initially tried using the mean (average), but a few extreme outliers caused Claude Code's cost to look far higher than it typically is.

Claude Code may simply be best used with Anthropic's models and quite bad with Kimi. An alternate solution is to remove Claude Code from the diagram if its so far off from the others that it causes scaling problems. I was looking at the results JSON and it looks like there is only one run of each task with each harness. Since these are disparate tasks using the median means that the headline cost is the cost of one s…

[flagged]

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#58

Post by benw214 »

I'm not sure i agree with these. I've been testing a simple prompt with different model and harness combinations to work out which one produces best results. I do this in '/goal' mode.

Prompt -

"Build a single-page Three.js sci-fi hangar with hovering drones, animated warning lights, emissive runway strips, and subtle volumetric-style fog planes. Include drone formation toggle and cinematic camera path. Output one self-contained HTML file with inline JavaScript."

You can see the outputs here - https://alvins82.github.io/hangar-harness-model-tests/ https://github.com/alvins82/hangar-harness-model-tests/

OMP i would say comes out on this prompt the best out of all the harnesses i've tried. I suspect possibly a lot of this is to do with the base system prompt that each harness injects.

Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x

#59

Post by billylo »

Edward40 wrote: post #53
markbao wrote: post #32

This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed. People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up. Thanks for making this and filling a real gap!

Thanks! V1.1 is set to add more harnesses and evaluate a broader range of models. We're moving to a full harness × model matrix to uncover interaction effects and release a complete harness-model compatibility map.

I am looking forward to your findings on home-field advantage. It's natural that Claude Code will work better with Anthropic models. If we know by how much, it will inform many of us.
Post reply on HN