There really should be error bars on those measurements. With just 30 samples, the 95% confidence intervals for accuracy should all be more than 35 percentage points wide, so comfortably overlap. For cost it's harder to say, because outcomes aren't constrained to {0, 1}, but I also expect a lot of variability there.
Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
31–40 of 67 posts
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#32This is great. People always talk about how important the harness is and yet we have so few harness benchmarks. Agree with sibling commenter that ‘harness x model’ is needed.
People always talk about how Cursor harness has some secret sauce; would like to see how that one stacks up.
Thanks for making this and filling a real gap!
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#33One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it.
It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#34One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…
I think it's still a useful data point. For example, omp, which is pi with some default extensions, scores worse. I do agree that adding more configurations of Pi would help though.
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#35[flagged]
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#36Forgive my stupidity but how do you run Claude Code with non-Anthropic models?
Here's an example for using it with GLM:https://docs.z.ai/devpack/tool/claude ... works similarly for other models
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#37Do you have instructions on how to run a custom harness against this? There are none on the linked page. I want to run Dirac ( https://github.com/dirac-run/dirac ).
Their code and evalations are linked to on the blog post accompanying the results:
Blog Post: https://runta.com/blog/introducing-frontierharness-eval/
The tasks and methodology: https://github.com/runta-dev/frontier-harness-eval
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#38Interesting to see harness benchmarks for coding agents. The same problem exists for conversational and data agents but I dont see anyone benchmarking them yet. Seems like manual-spot checks are the norm
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#39Is Github Copilot (integrated in VS Code) not a thing? I use it all the time and don't know what more I could wish for. Disclaimer: I don't let agents run on huge tasks for hours. Almost all tasks I give them are done in under 30 min.
I also feel Copilot is not getting the attention it should considering its user base and its importance in the enterprise context. There is a lot to be learned and I found https://news.ycombinator.com/item?id=49256057 (mitm copilot investigation) interesting. Playing with Copilot and asking it what was the biggest part of its standard payload I was surprised to learn that VSCode extensions plug in their MCP servers without me being much aware of it.
Re: Show HN: FrontierHarness Eval – 9 harness, same model, cost per pass varies 17x
#40One issue I see with including something like Pi is that it's intentionally bare bones. I don't think anyone uses Pi without some basic custom extensions (subagents, check lists, etc), so this benchmark may not be representative of a realistic setup. Would be interesting to see some sort of ablation test too. E.g. what parts of Codex contribute most to the perf, and can they be recreated more minimally in something l…
It does seem a bit odd to say "We tested all of these bicycles with the same rider" when one of the test cases is actually a bare high-end frame with no components on it. It's even weirder given that another of the test cases is essentially "We put parts on the frame (OhMyPi) and it went faster!"
I'm not sure your analogy holds, because here the optimization metric is clear and unanimous: every one wants max pass rate at min costs.
With the bicycle, some may prefer comfort, others speed, others offroad, etc., so it would not be obvious which one is "best".